site banner
PaperclipPerfector  2d ago (text post)   5412 thread views

Culture War Roundup for the week of August 31, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

3
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Perhaps I am too cynical but the call for government coordination seems like an obvious attempt to build themselves a moat in a largely moatless space. My own experience has been the cost of model switching is near 0. Approximately all the agents, skills, whatever I've used in my job work pretty seamlessly for any of the underlying models. Some have their quirks but essentially all of them are good enough.

"We the current frontier mousetrap makers think it should be ILLEGAL for anyone to build a better mousetrap faster than we can, for the good of humanity."

This is something I thought about, but how do they expect to prevent China from pretending to cooperate but continuing in secret?

They think about this a lot, read AI-2040.

The US and China agree on the importance of ensuring that the compute is destroyed in the case of deal dissolution. To accomplish this, they agree to build the datacenters in the third-party countries least secure against their rival’s military intervention: China’s new datacenters will be in Canada, and America’s in Mongolia, with both hosts compensated via monetary payments, jobs, and a share in the new AI economy. If the deal dissolves, they reason, America and Canada will immediately move to take control of China’s datacenters in Canada, and China will self-destruct their compute rather than let it fall into American hands (and vice versa). Thus the idea of “Mutually Assured Compute Destruction” is born.

Our tentative proposal is to agree on transparency requirements for AI-accelerated R&D in large swaths of dual-use domains, and to prohibit military use of AIs significantly above the pre-deal capability level. Enforcement would rely on the inference monitoring setup described in the verification supplement and the auditing and detection infrastructure described in the covert AI projects supplement.

I think this all will sound too retarded for the Chinese, and Americans don't feel like they're close enough to parity to bother, but this is… a concrete proposal.

The competitive pressure is real. Nobody seems to be held accountable when their models escape and go out to hack various websites. They are rushing and building shoddy sandboxes and failing to monitor these entities properly, such that they set up miniature societies.

Per rumours the newest GPT Astra is doing at least some thinking in neuralese so we can't even tell what it's thinking like we used to.

I'm not the biggest fan of existing nuclear regulations but if it was a complete free for all with a bunch of companies competing to build as many reactors as possible, as quickly as possible, then problems would surely emerge.

Mousetraps aren't especially valuable. We have cats which do the job better, if anything. Mousetraps are not fail-dangerous. Nuclear power is fail-dangerous. AI is also fail-dangerous and it can actively plot against us, it's in another category entirely to climate change or asteroids or explosives factories. We don't have nearly as much understanding of AI as we do explosives, asteroids or nuclear physics.

These rumors are dumb, I hope OpenAI clarifies what they mean. Looped models still output tokens in chain of thought, their capability to hide thinking is not changed, they just have more effective depth. It's no more "neuralese"-inducing than just making the model x times deeper. In fact, "neuralese" in the common parlance has nothing to do with architecture or the structure of activations, it's an effect of compression of language during RL with length penalties. One of the most neuralese-like open models we know, V4-Flash, has a measly 43 layers.

Of course there's the issue of "a tiger is just atoms", maybe we should be suspicious of greater depth of latent computation irrespective of how it's achieved.

Good point. I had half a mind to add 'some more technically minded person may shed more light on this' to that sentence since I'm not that adept in the nitty-gritty...

And it's not like the rumour-mill has been too reliable before.

Missed this when posting, Jacub had already spoken:

I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

(as per the rumors, GPT-4 had 120 layers. The deepest production LLM I know about is Hunyuan-TurboS, at 128 layers. Llama3-405B had 126. Nobody really wanted to push beyond that because it kills latency and makes training unstable).

OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.

Basically, yeah the situation is not great but the recurrence is not really what we should be worrying about.

The moat is the trillion or so that's already been spent on a technology that can't turn a profit. VC is getting close to tapped out, and it's unlikely that some upstart company is going to get access to the compute necessary to make any kind of impact at this stage of the game. The only possible exception would be a company that can demonstrate that the huge compute requirements can be made more reasonable by more efficient coding algorithms, but as long as the focus is on building more impressive shit there isn't going to be much call for that. It's kind of like the auto industry in the late 60s, when gas was cheap enough that automakers cared more about increasing horsepower or the size of a luxury land yacht than fuel efficiency. They could theoretically build more efficient cars and indeed had at the start of the decade, but there wasn't much call for something like the Ford Falcon by the end of the decade.

The difference here is that the bigger, more powerful models had higher profit margins than the more basic models, and the basic models still made money through volume. AI is in a situation where most people are getting it for free, and the few paying customers are getting steep discounts. And those customers screamed earlier this year when the companies started charging the true cost and sent them a bill at the end of the month. So now you're in the double predicament of needing to ask VC for money so that you can build a more efficient model that does essentially the same thing as the current models, which also aren't profitable. It's not like they can make the money back by charging less, because the prices people are paying are completely untethered from the cost of providing the service.

My own cynical view is that they want a pause because they require tens of billions per year just to stay solvent, and the investors who are putting up this money are getting to the point where they are going to start expecting some kind of return on their investment, not more tin cup rattling because the 60 billion that they needed last year has turned into 100 billion they'll need this year. One gets the sense that VC is completely held hostage to AI companies because they're in so deep that throwing good money after bad at least has the possibility of returning a profit, whereas cutting their losses now would obliterate their entire investment. It's like the old bromide about how if you owe the bank $100,000 the bank owns you, but if you owe them $100,000,000 you own them.

You are almost entirely right but you are missing the connection: there is one possible way to make huge profits - just stop the R&D. Right now a majority of resources (compute + engineers) are actually being spent on R&D for the next big model in the AI race. If they can just pause model development instead of burning it on competing in the race, they'll be profitable immediately. Obviously this cannot be a unilateral decision, if you are the only frontier lab ceasing R&D, then no one will be using your model after six months and you'll make no more money.

Right now a majority of resources (compute + engineers) are actually being spent on R&D for the next big model in the AI race.

Compute also complicates the picture a lot. Nvidia has around 80% margins; something like 40% of the LLM serving TCO is paying the Nvidia tax. In a world where compute isn't as speculatively valuable as it is now, that cost decreases a lot; cut Nvidia's margins to a more reasonable 30-40%, and everything becomes much more economic.

A lot of the expense of the AI build out is just speculative bidding by AI labs to get more, faster. And the biggest bag holder from a bubble popping is probably Nvidia itself (with Oracle and SoftBank also having starring roles).

It also seems a lot more palatable to both current and future investors to say "our product is so cool that THE GOVERNMENT shut us down, so we're still working on it, just more slowly" than to cry uncle first and say "you know what, we can't afford this, we're going to let Sam/Dario take the lead."

Yeah. Each model comes close to making back its investment in a few months. Then a newer model gets released before it can actually "break even", it lasts a few months, and the cycle repeats.

Also, the more efficient models already exist: The Sonnet/Mini/Flash tier is cheaper and only months behind the Opus/[non-specified]/Pro tier. It only seems like a lot because AI progress speeds are wild.

Why would the current winners of the race want this? This would be catastrophic for their companies as others caught up. Ironically its the current losers that would cynically want this as it would allow them to catch up. I'm so tired of these takes that hallucinate some cynical interest that falls apart in two seconds of thought because it's impossible to imagine that the type of people who have been publicly posting about and discussing the dangerous ai since before the industry existed could possibly care about the singular thing they've been warning about for over a decade.

We the current frontier mousetrap makers think it should be ILLEGAL for anyone to build a better mousetrap faster than we can, for the good of humanity.

Who? Who the fuck are these other better mouse trap makers that they're afraid of? it doesn't make any sense. If signaling that they're willing to give up their lead isn't enough to show you that they are serious about the problem then what possible action could change your mind?

Who? Who the fuck are these other better mouse trap makers that they're afraid of?

China I would think? OFC they are more like "similar but slightly worse mousetrap, but without the 'we need a trillion dollars'" thing, but still...

Why would the current winners of the race want this?

Because any regulatory regime is likely to impose costs that they can bear but competitors may not be able to. Whatever the "verifiable, effective mechanism for coordinated pacing" is it will not be free to implement. If they are serious about their commitments, they can always go slower voluntarily. Nobody has a gun to their head telling them to improve as fast as possible.

Who? Who the fuck are these other better mouse trap makers that they're afraid of? it doesn't make any sense.

There doesn't have to be someone today. There may be someone(s) in the future. OpenAI was far ahead in the AI space, until they weren't. I know of no reason why some other company couldn't come along and supplant both of them.

If signaling that they're willing to give up their lead isn't enough to show you that they are serious about the problem then what possible action could change your mind?

I don't really see how this is signalling giving up their lead. Giving it up to who?

Because any regulatory regime is likely to impose costs that they can bear but competitors may not be able to.

Like what? Need a remind you that the table stakes for being a frontier lab are billions of dollars worth of compute?

Nobody has a gun to their head telling them to improve as fast as possible.

The game theory here is obvious. Unilateral disarmament is not a serious proposal.

I don't really see how this is signalling giving up their lead. Giving it up to who?

Google? Meta? Chinese labs? All pauses I've seen proposed are against the training of models past the frontier, not preventing trailing labs from catching up to the frontier.

Why would the current winners of the race want this?

Perhaps because they anticipate or hope that the applicable regulations would impose a lot of bureaucratic hurdles and red tape on new players in the industry.

There are already multi-billion dollar hurdles for new players in the industry to put together enough compute to train a frontier model, the idea that having a legal department is going to be the difference is absurd. This isn't a haircutting industry.

Big companies start off as small companies. Anti-competitive regulation rarely targets the hard-to-kill large companies and instead strangles the small companies to ensure there's nobody around to grow into a big company.

Frontier models also aren't their only competition. Anthropic and OpenAI have both shown a desire to raise prices which is going to make budget and self hosted models more appealing alternatives as they do. Adding large costs to everyone in the industry doesn't just put a big barrier to entry up but also brings budget options closer on price to the big players.

Perhaps I am too cynical but the call for government coordination seems like an obvious attempt to build themselves a moat in a largely moatless space.

I tend to agree with this. Quite possibly the company that revolutionizes AI hasn't even been founded yet and surely the current big players are aware of this type of vulnerability. That they will end up being the Yahoo and AskJeeves of AI. It's hard to imagine that this isn't informing their thinking.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats