This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Oh, I absolutely think it's high enough to take seriously, I just don't think it's so high that the "regardless of whether you particularly like the future we propose, you should agree that it's at least somewhat better than the certain extinction that is the alternative and therefore support us" argument of team MIRI goes through. This of course does depend on your value function a lot, but in my eyes the expected value of "20% chance of unsafe AGI apocalypse" is higher than the expected value of MIRI's pivotal act timeline, which in turn is higher than the expected value of "100%-\varepsilon chance of unsafe AGI apocalypse". This ordering is what gives rise to the significance I assign to the "bean-counting" of ways in which the LW scenario could fail to come pass, since I really think the aggregate of individually unlikely scenarios an AGI could fail to take off can push the likelihood of that existential risk down into the 10^-1 range. I don't know if this is weird; I can see it being a consequence of myself having a comparatively (negative? misanthropic?) personality which makes me value highly misaligned but nominally "human" existence closer to complete nonexistence than to similar-to-present-day human existence. Certainly, someone with the right kind of anthropophilic outlook may instead consider human extinction so much worse than guaranteed continued human existence that is morally warped with no prospect of redemption that taking the 20% chance of extinction over the 100% chance of the latter seems barbarous.
I do care if I think there's a significant chance that it can't conceal its capabilities, and I think that 20 years from emergency to complete takeover is quite a plausible timeline too, since I'm really not sold on the "slightly smarter than humans on silicon substrate => many orders of magnitude faster improvement" belief.
Well, neither, but I think it's reasonably likely that the candidate for takeoff AGI will be military-adjacent as those applications are a competition sink far removed from civilian control and generally already endowed with spicy actuators. With those, though, it's quite likely that a generic response path geared towards MAD-disrupting superweapons will be triggered. Certainly, if I were Putin and my long-running uneasy stalemate in Ukraine started getting disrupted by game-changing NATO AI drone swarms, I'd be strongly considering the merits of forcing a future rematch under more favourable conditions via the global thermonuclear war route.
Yeah, I don't find that particular path to be likely for perpetual non-apocalypse; this is just saying that even if it will still take AGIs another hundred years to figure out how to improve and really leave us in the dust, we will grant them all the time they need. Instead, I'm betting on the "AGI takeoff will fizzle, resulting in chaos that destroys the technical preconditions for it for a long time" space.
Yeah, what I'm saying is that I find it quite likely that a budding AGI takeoff will result in the "end [at least temporary] of modern civilization". "Modern civilization", at least as needed to sustain the computational substrate for cutting-edge AGIs, seems quite fragile to me. An AGI could, with time, of course refine itself to be less brittle, but I suspect, as a consequence of believing self-improvement to be rather hard, that it would not manage to do that in time before disruption due to its other applications causes civilisational collapse.
More options
Context Copy link