This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
The future of AI will be dumber than we can imagine
Recently Scott and some others put out this snazzy website showing their forecast of the future: https://ai-2027.com/
In essence, Scott and the others predict an AI race between 'OpenBrain' and 'Deepcent' where OpenAI stays about 3 months ahead of Deepseek up until superintelligence is achieved in mid-2027. The race dynamics mean they have a pivotal choice in late 2027 of whether to accelerate and obliterate humanity. Or they can do the right thing, slow down and make sure they're in control, then humanity enters a golden age.
It's all very much trad-AI alignment rhetoric, we've seen it all before. Decelerate or die. However, I note that one of the authors has an impressive track record, foreseeing roughly the innovations we've seen today back in 2021: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like
Back to AI-2027! Reading between the lines, the moral of the story is for the President to centralize all compute in a single project as quickly as he can. That's the easiest path to beat China! That's the only way China can keep up with the US in compute, they centralize first! In their narrative, OpenAI stays only a little ahead because there are other US companies who all have their own compute and are busy replicating OpenAI's secret tricks albeit 6 months behind.
I think there are a number of holes in the story, primarily where they explain away the human members of the Supreme AI Oversight Committee launching a coup to secure world hegemony. If you want to secure hegemony, this is the committee to be on - you'll ensure you're on it! The upper echelons of government and big tech are full of power-hungry people. They will fight tooth and nail to get into a position of power that makes even the intelligence apparatus drool with envy.
But surely the most gaping hole in the story is expecting rational, statesmanlike leadership from the US government. It's not just a Trump thing - gain of function research was still happening under Biden. While all the AI people worry about machines helping terrorists create bioweapons, the Experts are creating bioweapons with all the labs and grants given to them by leading universities, NGOs and governments. We aren't living in a mature, well-administrated society in the West generally, it's not just a US thing.
But under Trump the US government behaves in a chaotic, openly grasping way. The article came out just as Trump unleashed his tariffs on the world so the writers couldn't have predicted it. There are as yet unconfirmed reports people were insider-trading on tariff relief announcements. The silliness of the whole situation (blanket tariffs on every country save Belarus, Russia, North Korea and total trade war with China... then trade war on China with electronics excepted) is incredible.
I agree with the general premise of superintelligence by 2027. There were significant and noticeable improvements from Sonnet 3.5, 3.6 and 3.7 IMO. Supposedly new Gemini is even better. Progress isn't slowing down.
But do we really want superintelligence to be centralized by the most powerhungry figures of an unusually erratic administration in an innately dysfunctional government? Do we want no alternative to these people running the show? Superintelligence policy made by whoever can snag Trump's ear, whiplashing between extremes when dumb decisions are made and unmade? Or the never-Trump brigade deep in the institutions running their own AI policy behind the president's back, wars of cloak and dagger in the dark? OpenAI already had one corporate coup attempt, the danger is clear.
This is a recipe for the disempowerment of humanity. Absolute power corrupts absolutely and these people are already corrupted.
Instead of worrying 95% about the machine being misaligned and brushing off human misalignment in a few paragraphs, much more care needs to be focused on human misalignment. Decentralization is a virtue here. The most positive realistic scenario I can think of involves steady, gradual progression to superintelligence - widely distributed. Google, OpenAI, Grok and Deepseek might be ahead but not that far ahead of Qwen, Anthropic and Mistral (Meta looks NGMI at this point). A superintelligence achieved today could eat the world but by 2027, it would only be first among equals. Lesser AIs working for different people in alliances with countries could create an equilibrium where no single actor can monopolize the world. Even if OpenAI has the best AI, the others could form a coalition to stop them scaling too fast. And if Trump does something stupid then the damage is limited.
But this requires many strong competitors capable of mutual deterrence, not a single centralized operation with a huge lead. All we have to do is ensure that OpenAI doesn't get 40% of global AI compute or something huge like that. AI safety is myopic, obsessed solely with the dangers of race dynamics above all else. Besides the danger of decentralization, there's also the danger of losing the race. Who is to say that the US can afford to slow down with the Chinese breathing down their neck? They've done pretty well with the resources available to them and there's a lot more they could do - mobilizing vast highly educated populations to provide high-quality data for a start.
Eleizer Yudkowsky was credited by Altman for getting people interested in AGI and superintelligence, despite OpenAI and the AI race being the one thing he didn't want to happen. Really there needs to be more self-awareness in preventing this kind of massive self-own happening again. Urging the US to centralize AI (which happens in the 'good' timeline of AI-2027 and would ensure a comfortable lead and resolution of all danger if it happened earlier) is dangerous.
Edit: US secretary of education thinks AI is 'A1': https://x.com/JoshConstine/status/1910895176224215207
There are some problems with AI-2027. And the main argument for taking it seriously, Kokotaljo's prediction track record, given that he's been in the ratsphere at the start of the scaling revolution, is not so impressive to me. What does he say concretely?
Right from the start:
In reality: by August 2022, GPT-4 finished pretraining (and became available only on March 14, 2023), it used only images, with what we today understand was a crappy encoder like CLIP and projection layer bottleneck, and the main model was pretrained on pure text still. There was no – zero – multimodal transfer, look up the tech report. GPT with vision only really became available by November 2023. The first seriously, natively multimodal-pretrained model is 4o which debuted in Spring 2024. Facebook was nowhere to be seen and only reached some crappy multimodality in production model by Sep 25, 2024. “bureaucracies/apps available in 2022” also didn't happen in any meaningful sense. So far, not terrible, but keep it in mind; there's a tendency to correct for conservatism in AI progress, because prediction markets tend to overestimate difficulty of some benchmark milestones, and here I think the opposite happens.
Again, nothing of the sort happened, the guy is just rehashing Yud's paranoid tropes that have more similarity to Cold War era unactualized doctrines than any real world business processes. GPT-4 was on the order of $30M–$100M, took like 4 months, and was by far the biggest training run of 2022-early 2023, it was a giant MoE (I guess he didn't know about MoEs then, even though Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer is from 2017, same year as Transformer, from an all-star DM team; incidentally the first giant sparse Chinese MoE was WuDao, announced on January 11, 2021, it was dirt cheap and actually pretrained on images and text).
Notice the absence of Anthropic or China in any of this.
By the end of 2024, models were in training or pre-deployment testing that exceeded 3e26 FLOPs, and it still didn't reach $100M of compute because compute has been getting cheaper. GPT-4 is like 2e25.
I am not sure what he had in mind in this whole section on chip wars. China can't meaningfully retaliate except by controlling exports of rate earths. Huawei was never bottlenecked by chip design, they could leapfrog Nvidia with human engineering alone if Uncle Sam let them in 2020. There have been no noteworthy new players in fabless and none of new players used AI.
None of this happened, in fact China has rolled up more stringent regulations than probably anybody to label AI-generated content and seems quite fine with its archaic methods.
This is not at all what we ended up doing, this is a cringe Lesswronger's idea of a way to build a reasoning agent that has intuitive potential for misalignment and adversarial manipulative stance towards humans. I think Noam Brown's Diplomacy work was mostly thrown out and we returned to AlphaGo style of simple RL with verifiable rewards from math and code execution, as explained by DeepSeek in R1 paper. This happened in early 2023, and reached product stage by Sep 2024.
We've caught up. I think none of this looks more impressive in retrospect than typical futurism, given the short time horizon. It's just “here are some things I've read about in popular reporting on AI research, and somewhere in the next 5 years a bunch of them will happen in some kind of order”. Multimodality, agents – that's all very generic. “bureaucracies” still didn't happen, this looks like some ngmi CYC nonsense, but coding assistants did. Adversarial games had no relevance; annotation for RLHF, and then pure RL – had. It appears to me that he was never really fascinated by the tech as such, only by its application to the rationalist discourse. Indeed:
OK.
Now as for the 2027 version, they've put in a lot more work (by the way Lifland has a lackluster track record with his AI outcomes modeling I think, and also depends in his sources on Kotra who just makes shit up). And I think it's even less impressive. It stubbornly, bitterly refuses to update on deviations from the Prophecy that have been happening.
First, they do not update on the underrated insight by de Gaulle: “China is a big country, inhabited by many Chinese.” I think, and have argued before, that by now Orientals have a substantial edge in research talent. One can continue coping about their inferior, uninventive ways, but honestly I'm done with this, it's just embarrassing kanging and makes White (and Jewish) people who do it look like bitter Arab, Indian or Black Supremacists to me. Sure, they have a different cognitive style centered on iterative optimization and synergizing local techniques, but this style just so happens to translate very well into rapidly improving algorithms and systems. And it scales! Oh, it scales well with educated population size, so long as it can be employed. I've written on the rise of their domestic research enough in my previous unpopular long posts. Be that as it may, China is very happy right now with the way its system is working, with half a dozen intensely talented teams competing and building on each other's work in the open, educating the even bigger next crop of geniuses, maybe 1 OOM larger than the comparable tier graduating American institutions this year (and thanks to Trump and other unrelated factors, most of them can be expected to voluntarily stay home this time). Smushing agile startups into a big, corrupt, centralized SOE is NOT how “CCP wakes up”, it's how it goes back to its Maoist sleep. They have a system of distributing state-owned compute to companies and institutions and will keep it running but that's about it.
And they are already mostly aware of the object level; they just don't agree with Lesswong analysis. Being Marxists, they firmly believe that what decides victory is primarily material forces of production, and that's kind of their forte. No matter what wordcels imagine about Godlike powers of brains in a box in a basement, intelligence has to cash out into actions to have effect on the world. So! Automated manufacturing, you say? They're having a humanoid robot half-marathon in… today I think, there's a ton of effort going into general and specialized automation and indinegizing every part of the robotic supply chain, on China scale that we know from their EV expansion. Automated R&D? They indinegize production of laboratory equipment and fill facilities. Automated governance? Their state departments compete in integration of R1 already. They're setting up everything that's needed for speedy takeoff even if their moment comes a bit later. What does the US do? Flail around with alienating Europeans and vague dreams of bringing 1950s back?
More importantly, the authors completely discard the problem that this work is happening in the open. This is a torpedo into Lesswrongian doctrine of an all-conquering singleton. If the world is populated by a great number of private actors with even subpar autonomous agents serving them, this is a complex world to take over! In fact it may be chaotic enough to erase any amount of intelligence advantage, just like longer horizon on weather prediciton sends the most advanced algorithms and models to the same level as simple heuristics.
Further, the promise of the reasoning paradigm is that intrinsically dumber agents can overcome problems of the same difficulty as top-of-the-line ones, provided enough inference compute. This blunts the edge of actors with the capital and know-how for larger training runs, reducing this to the question of logistics, trading electricity and amortized compute cost for outcomes. And importantly, this commoditization may erase the capital that “OpenBrain” can raise for its ambition. How much value will the wealthy of the world part with to have stake in the world's most impressive model for a whole of 3 months or even weeks? What does it buy them? Would it not make more sense to buy or rent their own hardware, download DeepSeek V4/R2 and use the conveniently included scripts to calibrate it for running your business? Or is the idea here that OpenBrain's product is so crushingly superior that it will be raking billions and soon trillions in inference, despite us seeing already that inference prices are cratering even as zero-shot solution rates increase? Just how much money is there to be made in centralized AI, when AI has become a common utility? I know that not so long ago the richest guy in China was selling bottled water, but…
Basically, I find this text lacking both as a forecast, and on its own terms as a call to action to minimize AI risks. We likely won't have a singleton, we'll have a very contested information space, ironically closer to the end of Kokotaljo's original report, but even more so. This theory of a transition point to ASI that allows to rapidly gain durable advantage is pretty suspect. They should take the L on old rationalist narratives and figure out how to help our world better.
Do you have any sources/context for technical criticisms like this, so that those of us who haven't closely followed AI development can better understand your criticism? I know 3e26>5e25, but not whether "a single training run" and "training or pre-deployment testing" are comparable or if "$100M of compute" is a useful unit of measure.
I am not sure how to answer. Sources for model scales, training times and budgets are part from official information in tech reports, part rumors and insider leaks, part interpolation and extrapolation from features like inference speed and pricing and limits of known hardware, SOTA in more transparent systems and the delta to frontier ones. See here for values from a credible organization..
$100M of compute is a useful measure of companies' confidence in returns on a given project, and moreover in their technical stack. You can't just burn $100M and have a model, it'll take months, and it practically never makes sense to train for more than, say, 6 months, because things improve too quickly and you finish training just in time to see a better architecture/data/optimized hardware exceed your performance at a lower cost. So before major releases people spend compute on experiments validating hypotheses and on inference, collect data for post-training, and amass more compute for a short sprint. Thus, “1 year” is ludicrous.
Before reasoning models, post-training was a rounding error in compute costs, even now it's probably <40%. Pre-deployment testing depends on company policy/ideology, but much heavier in human labor time than in compute time.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link