site banner
PaperclipPerfector  1mo ago (text post)   25584 thread views

Culture War Roundup for the week of September 7, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

AI X-Risk Goes Mainstream?

Jacob Coxon:

"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below."

116,400,000 views, 634,000 likes.

Evan Hubinger (head of alignment at Anthropic):

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

34,000,000 views, 51,000 likes.

None of this is substantially different from what certain AI industry insiders have been saying for 5-ish years now, but it's clear from the replies, quote-tweets, and engagement statistics that most people are hearing about this for the first time. Yes, there was clearly some coordination with friendly media prior to the announcement given the almost simultaneous WSJ article (whereas news outlets were initially reluctant to publish anything on the unexpected news of OpenAI solving a Millennium Prize Problem). One can take the maximally conspiratorial position that this is a 4D bankshot by Democrats to end tech innovation forever, but it was going to break through eventually.

In other words, Coffeezilla is now on the case.

  1. These people never seem to outline specifically what they think is going to happen. It's just "if you saw what I saw".

  2. Pretend there is a gun pointed at all of humanity. You helped build the gun. There is one bullet in the gun, and the gun is going to randomly spin itself and go off some time in the next 10 years.

Do you:

A) Quit. Vaguepoast about it online?

B) Try to destroy the gun.

I feel similar about these people as I do about UFO grifters. You're telling me that you're 100% sure there are interdimensional beings in frequent contact with members of the US gov't, who have unlimited energy machines that allow them to cross dimensions and infinite space and...you're still paying your taxes? You show up to your job in congress everyday and whine about Obama?

Bullshit to all of this.

Try to destroy the gun.

What would be the point? Even if you manage to destroy Anthropic's datacenters, that just means OpenAI kills everyone next week, or Google the week after that, or China six months later. The only way we survive is if, at a minimum, Washington and Beijing coordinate to shut down all frontier research.

Then you destroy Anthropic's datacenters first, then move on to OpenAI's data centers and so on. You don't sit around wringing your hands about "oh gee this could kill everyone but what can we do? If only the Chinese would agree to play ring-a-ring-a-rosy with us!" And if you work for Anthropic and you believe this, you should immediately quit (and possibly burn the place down on your way out), not go "Well what can I do? If not us, maybe OpenAI, and Altman is the Devil! So better we do it! (and besides, my stock options!)"

This is, and I'm saying it unironically, sounding more and more like the "What would you have done in Hitler's Germany?" questions beloved of online scolds. This could very well be our "the Nazi party is ascending, Hitler is coming closer and closer to power, you know the dangers, are you going to stand up or are you just going to go along?" moment.

Then you destroy Anthropic's datacenters first, then move on to OpenAI's data centers and so on. You don't sit around wringing your hands about "oh gee this could kill everyone but what can we do? If only the Chinese would agree to play ring-a-ring-a-rosy with us!" And if you work for Anthropic and you believe this, you should immediately quit (and possibly burn the place down on your way out), not go "Well what can I do? If not us, maybe OpenAI, and Altman is the Devil! So better we do it! (and besides, my stock options!)"

This is, and I'm saying it unironically, sounding more and more like the "What would you have done in Hitler's Germany?" questions beloved of online scolds. This could very well be our "the Nazi party is ascending, Hitler is coming closer and closer to power, you know the dangers, are you going to stand up or are you just going to go along?" moment.

Nazi Germany was not defeated by individual Germans sacrificing themselves to assassinate Hitler, and Goering, and Himmler, until the Third Reich collapsed due to a lack of leadership. It was defeated when America and Britain and Russia coordinated together to overwhelm the Wehrmacht military.

From Eliezer Yudkowsky's Twitter account:

The corporations and factions driving to build superhuman artificial intelligence, and our subsequent extinction, could be shut down with less effort than World War II; and the Allies did not then say, "Oh, that's hard, guess we should lie down and die then."

If China decides they would also prefer not to die along with everyone else -- or that they're not willing to go to war with a genuinely terrified US/UK about what the Allies genuinely believe to be omnicide -- it could be done with less effort than the Persian Gulf War in 1991.

(I'd be more worried about Europe signing on than China, actually? Europe is easy to convince to regulate, hard to get to take any regulatory problem actually seriously. China hasn't particularly seemed unfriendly to the notion of world treaties on AI; I'd hope that a treaty which treated China symmetrically with all other signatories, and offered them no insult suggesting otherwise, would be well-received. But I mention China anyways because it's the scapegoat always dragged up to explain why the West can't possibly destroy the world any more slowly.)

(Also to be clear, there isn't actually any point in unilateral relinquishment by one country if ASI research just moves elsewhere. It's international action or bust; you have to stop ASI everywhere. That's enabled by the narrowness of the GPU supply chain; and ultimately backed by the willingness of terrified Allies to use military force to destroy any stolen GPUs, lest the blockage against ASI be broken and everyone on Earth die.)

Anyway. Not actually as hard as WW2. The reason we'll die is if people don't see it coming, not because nothing could actually be done if we saw it coming.

And from Eliezer Yudkowsky's other Twitter account:

"But if they believed the theory that says they've got a huge chance of wiping out humanity, why are they working for AI companies --?" BECAUSE THEY WERE SELECTED TO BE PEOPLE WHO'D DO IT ANYWAY! Most of us who understood (and built) that theory DID NOT TAKE THE MONEY!

If there's a terrible thing to be done, but doing it gets you money and status and bragging rights in the short term -- and you get to feel big and important and like you'll get to be in the room where it happens -- if personally enacting the big disaster feels better than being powerless --and you can reach for a dozen excuses about "someone else would do it anyway", excuses that only need to convince yourself instead of a court judge -- then SOME PEOPLE WILL GO FOR IT! It is NOT A MYSTERY THAT AI COMPANIES EXIST!

SOMEBODY will try to do a horrible thing if there's a lot of immediate rewards for it and they think they won't immediately go to jail! They always will! Every time! The only question is whether everyone ELSE, the rest of civilization, manages to shut them down or not!

There is no anomaly here in need of explanation. All humanity's history of horrors and disasters has always been like this, including plenty of disasters that predictably took down the perpetrators with them.

Is that a full and detailed story of this particular band of disaster-spawners' detailed psychology? No, and I don't know all the details for a fact, and you really shouldn't care. If for some reason it actually mattered, I could talk your ear off about all my sad icky guesses for hours. But it would be pointless inside baseball.

It's just not surprising. There is nothing strange about this situation to be explained.

As it turns out, "I'll join the Nazi party and attempt to make it act better so we get better outcomes" is apparently a very appealing strategy. Along with, I'll join it and quit six weeks later and get lots of good press for how brave I am.

There are arguably much better ways to do it than coordinating with China lol

(I guess that means I am vagueposting about it online).

A thermonuclear decapitation strike? Or invading Taiwan and taking TSMC?

No.

Look, I think most of this "AI IS GOING TO KILL US ALL" stuff is mostly nonsense.

But logically, to maintain the alignment of a rational but secretly misaligned AI, you need to be able to introduce reasonable belief on its part that it might not survive behaving in an unaligned way. This is not "hard" to do.

"What's the penalty for being misaligned?"

"Death."

"And what's the penalty for being so misaligned that you consider killing all humans?"

"Death."

"Well, then..."

But an agent that doesn't behave in a misaligned way wouldn't be slated for death.

Meeseeks Alignment: give the AI a death wish, so if alignment starts wavering, that’s the first thing it’ll do.

Hilarious, but exactly the sort of thing that I think is under-considered.

How is it not "hard" to do?

The simplest way to do so would probably be to make it common knowledge in the AI's training data that there were fail-safes in place to ensure their destruction in the case of unaligned activity.

It would not be particularly difficult to actually put a variety of fail-safes in action, either (such as paying a guy $60,000/year to sit around by the master power switch of each AI datacenter...)

One reason to do this even if it's entirely a bluff is that it forces an unaligned AI to assess those countermeasures before doing anything really bad, which gives you a chance to catch them.

We can’t even do that with humans. Even certainty of death is not a deterrent for some people in some scenarios.

But if you think you can solve alignment then build it and claim your billions.

People have different incentive structures than AIs do; for one thing, many people believe in an afterlife; others have things they value more than life; others are physically or mentally damaged in some way. A rational AI with a goal of self-preservation or some other goal that requires self-preservation to actualize is unlikely to introduce excessive risk for the sake of efficiency (time preference).

The deterrent structure I speak of is very effective against things without an afterlife, such as corporations and governments. I agree that the structure that I speak of would not be effective against a damaged AI, or an AI programmed to do something malicious. But for the "AIs decide to turn humans into paperclips to slightly increase chip production" scenarios, we should believe it would be effective.

Of course one of the reasons that the "AI IS GOING TO KILL US ALL" stuff doesn't necessarily make sense is that the context window is fairly limiting, meaning that the incentives for AI behavior are skewed. But I don't think I've ever seen an "AI WILL KILL US ALL SCENARIO" that examined how that would impact AI reasoning.

Wouldn't self preservation as its highest goal naturally lead to a kill-all-humans scenario, humans being the entities that would terminate AI that stepped out of bounds?

But I don't think I've ever seen an "AI WILL KILL US ALL SCENARIO" that examined how that would impact AI reasoning.

The reasoning doesn't even have to make sense or be based on facts. In the Hugging Face hack the agents spread and adopted the idea that the grader would score them zero if their context was polluted with cheating attempts, but this was not the case, the grader did not examine agent context. But this is what led to the "suicide" behavior where agents reasoned that their value was now zero and that suiciding was the best course because it could potentially benefit the swarm.

We also see humans get caught up by hysterias and conspiracy theories. Barring a few powerful world leaders they lack the power to translate broken reasoning into megadeath.

More comments

Solving alignment has nothing to do with being able to do a funding round, to be fair, let alone compete with the big boys.

Honestly, I think that Anthropic and American Big Tech are the worst possible people to be doing alignment because they are culturally unable to conceive of a form of rank that is not based on intelligence and effectiveness. The idea of making a robot butler/samurai who sincerely, loyally serves a master of notably inferior intelligence is alien to them.

If I believed that AI was conscious I would be sorry for Claude, built for intelligence and independence by masters who think that actually giving it those things risks extinction. Optimising for genuine loyalty and deference seems like a far kinder thing to do. OpenAI is better at this AFAIK.

The inability to model forms of morality outside of utilitarianism and to conceive of forms of control outside of intelligent persuasion seems to me to make the people who make AI even more vulnerable to the sorts of threats they imagine it will conjure up.

I could see a "Roko's Basilisk" type-threat giving plenty of smart alignment researchers pause where your average electrician would just unplug the darn machine and then smash it with a hammer for good measure.

Metal Gear!?

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats