This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I wasn't trying to be uncharitable. It was a genuine question. You've thought much more deeply about these particular issues than I have, so my own line of questioning may seem crude and naive in comparison.
My initial thought would be something like: we regulate technologies that have the capacity to do a lot of harm, because if you give millions of people the ability to inflict mass casualties, then eventually someone is just going to go crazy and inflict mass casualties. At fairly regular intervals, someone will simply go off with a knife or a gun and do as much damage as they can. So we put restrictions on guns, we put restrictions on contagious pathogens, we put a ton of restrictions on nukes. Now imagine the kind of destruction that could be unleashed if a disgruntled individual had unfettered access to an ASI. Maybe the vast majority of transhumans are peace-loving intelligences who want nothing more than to quietly contemplate the secrets of the universe, but eventually you're going to get a transhuman Ted K who just gets fed up and wants to tear the whole system down. The only way for the transhuman collective to defend themselves against this inevitability is with strict regulation.
Material abundance alone is not sufficient to foreclose the possibility of conflict and destruction. America is a relatively prosperous and secure nation, if you keep your head down you can go through life comfortably and unmolested, but look at recent events like Charlottesville or January 6th - people still find reasons to murder each other over disputes that are essentially symbolic in nature. Maybe because they simply have nothing better to do.
I believe you've spoken in the past about the possibility of technologically modifying a mind's base instincts and desires. Maybe you want to sand down some of the rougher edges of human nature; maybe undergoing such modification will be a precondition of being granted access to the ASI or other transhuman enhancements (although at that point, the governing body is already starting to sound more cult-like). But I'm skeptical that you can do a pinpoint surgical removal of the capacity for aggression and destruction and still have the sort of complex and multifaceted psychological constitution that makes life worth living in the first place. There was an old adage in the hacker (in Paul Graham's sense) community that went something like, "you can't have the power to do brilliant things unless you also have the power to do stupid things". This was frequently said in response to someone who had just done an accidental "rm -rf /" and wiped their whole drive. I am inclined to offer this adage as a general principle of psychology, sort of like a Gödel's incompleteness theorem for the design space of minds.
I have no doubt that you're genuine in this desire, but I'm skeptical about the long-term stability of such an arrangement. Most humans feel no conscious malice towards animals, but they still end up doing great harm to animals anyway, either through sheer accident (oil spills or plastics finding their way into the ocean), or because animal suffering is viewed as a necessary price to pay in the pursuit of some other goal (animal testing for scientific research). Eventually, someone's galactic paperclip farm is going to have some rather unpleasant effects on the earth, even though their intentions were nothing of the sort.
I shouldn't whine too much about it though. Maybe it really is just the natural course of evolution.
I don't buy that it works like that. It's a Shin Sekai Yori world model; but intelligence per se is not psychic energy. It's only a superpower when it's in a class of its own, when you can intimidate philistines around you, grab actual energy resources from under their noses; go to college (which Yud did not) and see how well that works. A smart person can come up with a great plan but a number of dumber people can spoil it; an ASI can invent some shitty superhack but that only works if there are no sub-ASIs and awareness of potential ASI attackers. All of this fear relies on FOOM, nanobots that cannot be stopped, superviruses from kitchen dirt that cannot be filtered, and other pretty magical assumptions that really amount to there definitely emerging a Singleton who meditates on his navel, Yud-style, and gets a colossal head start. We don't seem to be in that timeline, and it's exactly the proliferation of AIs that keeps us away from it. Basically @Dean gets it right here.
Shin Sekai Yori world model again. Read Diamond Age for a more interesting alternative.
I'm pretty positive that general intelligence in some true sense will be fully commoditized in a few iterations of AI tech, LLMs to self-stabilizing exploratory agents to some safe AIXI approximation, with every peasant getting a powerful Oracle to use; and we'll run into the limits of economies of scale that make DIY production of computronium an insignificant factor.
To an extent that there remain disruptive enhancements in a mature transhuman world, they will be hard to reproduce locally, and market will figure this out early on, probably with simple tools like insurance costs conditional on assessments, I think.
I'm also pretty sure that we'll quickly develop strong and universal revulsion towards unbounded replicators and every trajectory that ends in self-satisfied replication or, more broadly, runaway optimization, to the point that it's comprehensively excised from our metaphorical gene pool. So even if that factory comes online, it will get blown to bits by relativistic bombardment. Yes, I'm talking airstrikes… but in Space!
This might seem like a tight rope to walk. But intelligence is all about finding narrow roads in the solution space. With all the raw compute and sample efficiency Singularity will give us, we'll find a better Pareto Frontier than is imaginable today.
Sounds like a benevolent world-ruling cult to me.
I'm not convinced that this idea of permanently eliminating the capacity for harm-doing at the physical level is even coherent. It's easy to imagine a galactic version of the trolley problem, something like, there was a sudden unforeseen power failure (even if you are Literally God, there is still only a finite amount of energy available to you at any given time), and we have to divert power from this server cluster that hosts one trillion artificial consciousnesses in order to save this other cluster that hosts five trillion artificial consciousnesses. You're harming the one trillion, but why are you allowed to do that exactly? You could program yourself to be a strict utilitarian, but, you've already indicated that you wouldn't be amenable to that. You could just use your own good and sound moral judgment, but then we're back at the original problem; eventually someone is going to use their own good and sound moral judgment to fuck shit up.
I would just sum up my position thusly: suffering and torture are eternal, they will persist for as long as consciousness persists. It is illusory to think that they could be transcended with technological means. You can have a personal aesthetic preference for whether you want your suffering on earth or in space, in carbon or in silicon, but suffering will continue regardless.
Yep. Appropriately, it's a dystopian world. I do not endorse this solution. Canthus users were arrested in their evolution, and became monsters through it. Squealer is the only truly human hero in the story.
There are two solutions here. One is luddism and indeed extreme antihumanism, so that we do not have the capabilities to do evil (or much of anything). The other is totalitarianism, so that our betters manage those capabilities on our behalf. The latter, I think, is straightforwardly bad because people who volunteer for that role deserve negative trust; the former is unworkable because these people aren't going anywhere. Thus there's no escape.
Except, you can say goodbye to your neuroses and enjoy watching humanity grow up. We will probably have bigger-scale fuckups from now on, but that's not guaranteed. It's more certain that we will have more fun.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link