This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
The future of AI will be dumber than we can imagine
Recently Scott and some others put out this snazzy website showing their forecast of the future: https://ai-2027.com/
In essence, Scott and the others predict an AI race between 'OpenBrain' and 'Deepcent' where OpenAI stays about 3 months ahead of Deepseek up until superintelligence is achieved in mid-2027. The race dynamics mean they have a pivotal choice in late 2027 of whether to accelerate and obliterate humanity. Or they can do the right thing, slow down and make sure they're in control, then humanity enters a golden age.
It's all very much trad-AI alignment rhetoric, we've seen it all before. Decelerate or die. However, I note that one of the authors has an impressive track record, foreseeing roughly the innovations we've seen today back in 2021: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like
Back to AI-2027! Reading between the lines, the moral of the story is for the President to centralize all compute in a single project as quickly as he can. That's the easiest path to beat China! That's the only way China can keep up with the US in compute, they centralize first! In their narrative, OpenAI stays only a little ahead because there are other US companies who all have their own compute and are busy replicating OpenAI's secret tricks albeit 6 months behind.
I think there are a number of holes in the story, primarily where they explain away the human members of the Supreme AI Oversight Committee launching a coup to secure world hegemony. If you want to secure hegemony, this is the committee to be on - you'll ensure you're on it! The upper echelons of government and big tech are full of power-hungry people. They will fight tooth and nail to get into a position of power that makes even the intelligence apparatus drool with envy.
But surely the most gaping hole in the story is expecting rational, statesmanlike leadership from the US government. It's not just a Trump thing - gain of function research was still happening under Biden. While all the AI people worry about machines helping terrorists create bioweapons, the Experts are creating bioweapons with all the labs and grants given to them by leading universities, NGOs and governments. We aren't living in a mature, well-administrated society in the West generally, it's not just a US thing.
But under Trump the US government behaves in a chaotic, openly grasping way. The article came out just as Trump unleashed his tariffs on the world so the writers couldn't have predicted it. There are as yet unconfirmed reports people were insider-trading on tariff relief announcements. The silliness of the whole situation (blanket tariffs on every country save Belarus, Russia, North Korea and total trade war with China... then trade war on China with electronics excepted) is incredible.
I agree with the general premise of superintelligence by 2027. There were significant and noticeable improvements from Sonnet 3.5, 3.6 and 3.7 IMO. Supposedly new Gemini is even better. Progress isn't slowing down.
But do we really want superintelligence to be centralized by the most powerhungry figures of an unusually erratic administration in an innately dysfunctional government? Do we want no alternative to these people running the show? Superintelligence policy made by whoever can snag Trump's ear, whiplashing between extremes when dumb decisions are made and unmade? Or the never-Trump brigade deep in the institutions running their own AI policy behind the president's back, wars of cloak and dagger in the dark? OpenAI already had one corporate coup attempt, the danger is clear.
This is a recipe for the disempowerment of humanity. Absolute power corrupts absolutely and these people are already corrupted.
Instead of worrying 95% about the machine being misaligned and brushing off human misalignment in a few paragraphs, much more care needs to be focused on human misalignment. Decentralization is a virtue here. The most positive realistic scenario I can think of involves steady, gradual progression to superintelligence - widely distributed. Google, OpenAI, Grok and Deepseek might be ahead but not that far ahead of Qwen, Anthropic and Mistral (Meta looks NGMI at this point). A superintelligence achieved today could eat the world but by 2027, it would only be first among equals. Lesser AIs working for different people in alliances with countries could create an equilibrium where no single actor can monopolize the world. Even if OpenAI has the best AI, the others could form a coalition to stop them scaling too fast. And if Trump does something stupid then the damage is limited.
But this requires many strong competitors capable of mutual deterrence, not a single centralized operation with a huge lead. All we have to do is ensure that OpenAI doesn't get 40% of global AI compute or something huge like that. AI safety is myopic, obsessed solely with the dangers of race dynamics above all else. Besides the danger of decentralization, there's also the danger of losing the race. Who is to say that the US can afford to slow down with the Chinese breathing down their neck? They've done pretty well with the resources available to them and there's a lot more they could do - mobilizing vast highly educated populations to provide high-quality data for a start.
Eleizer Yudkowsky was credited by Altman for getting people interested in AGI and superintelligence, despite OpenAI and the AI race being the one thing he didn't want to happen. Really there needs to be more self-awareness in preventing this kind of massive self-own happening again. Urging the US to centralize AI (which happens in the 'good' timeline of AI-2027 and would ensure a comfortable lead and resolution of all danger if it happened earlier) is dangerous.
Edit: US secretary of education thinks AI is 'A1': https://x.com/JoshConstine/status/1910895176224215207
Eliezer Yudkowsky has explicitly noted* the alternative solution to this problem:
If you think China is going to destroy the world, the correct solution is not to destroy the world yourself as if RL is a game of DOTA; it's to stop China from destroying the world. Tell them that doing this will end the world. If they keep doing it, tell them that if they don't stop, you'll nuke them, and that their retaliation against this is irrelevant because it can't kill more Americans than the "all of them" that will be killed if they continue. If they don't stop after that, nuke them, and pray that there's some more sanity the next time around.
*To be clear, I was nearly done writing a similar essay myself, because I didn't think he had the guts to spit it out (certainly most top Rats don't). Apparently he did.
The race to develop bioweapons for absolutely no plausible reason continues, even after megadeaths. The AI race is locked in, it's staggeringly naïve to think this genie can be put in the bottle now. AI is profits and power, it's wildly popular and well-used. It's Eliezer and co up against Microsoft, Google, Facebook, DARPA, DoD, ERPers on /g/, kids who don't want to do their homework, a horde of ambitious entrepeneurs around the world, Tencent, Huawei, Nvidia and the Chinese state. Instant loss.
The US can't tell China 'stop or we nuke', they are themselves doing it right now bigger and better than anyone else! It's not framed as a race, it is a race and was always going to be a race. Politicians are uber-boomers and don't have the balls to go all in on anything like this with their lives, reputations and eternal legacy. They can't be totally sure that ASI means death. You only find out that ASI means death when it kills you.
I'd prefer this technology not to be developed at all. It's a terrible decision on a species-level. We spent hundreds of thousands of years wiping out our siblings in the Homo genus, we earned our Sapiens title and sole dominance of the world. Now we want to introduce a new contestant? Are we insane? But the dynamics demand it. When considering the balance of powers involved, the most Yud and co can hope for is to smooth out the edges a little bit, don't go all in on a strategy with 0 chance of success.
From where I sit, hoping for neural net alignment is itself a strategy with ~0 chance of success. Reality is under no obligation to give you a "reasonable" solution.
It's easier to see 'somehow we get ASI working for us or some subgroup of us' than 'everyone agrees to halt the race for power and profit and actually does so'. Not once in history has everyone stopped in a race for power and profit like this. Human cloning never even got off the ground, nobody was lobbying for it.
SALT/START seem like examples, albeit minor ones due to the difficulty of winning a nuclear war.
Also, I should note that a decent chunk of my P(~Doom) routes through WWIII removing AI as a thing-in-being-with-entrenched-interests.
Ending the nuclear arms race saved money and reduced costs, plus it's innately obvious that stacking up 50,000 H-bombs the great powers psy-opped into being unusable 99% of the time is somewhat dangerous and over the top. ASI is different, it's anywhere, anytime, for any purpose.
I think you underestimate the power of the forces behind AI. Everyone interested in technology wants it. Even if there's a full nuclear exchange, the surviving great powers and mega-corps will work far harder to achieve it and secure the strength and security that they so clearly need. The entrenched interests aren't so much Microsoft and DARPA but anyone with wealth, technology and a desire for power. WW3 going ugly would reduce the first two but ramp up the latter massively.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link