This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
A Big Day in the Culture War
Apologies for any incoherency or grammatical errors; today's events led me to down more than my usual share of booze.
Earlier today, big things happened in math. No, not Claude providing a sub quadratic 3SUM. OpenAI released hundreds of notable math results, in a GitHub repo.
Fun results:
Hilbert's tenth problem over (\mathbb{Q}) is undecidable
The quasi-Riemann hypothesis
The rational Hodge conjecture holds for every CM abelian variety
Integer multiplication can be done in sub-log-linear time. Oh, there's also a sub-log linear DFT.
Thoughts and observations:
Mathematicians are big mad. View the relevant subreddit on our progenitor site. Most there are probably, at best, adjuncts at community colleges desperately coping with the downward trajectory of already marginal careers, but it's fair to say that the writing is on the wall for mathematicians. There's probably a double digit number of grad students staring into a glass of whiskey tonight and thinking of hanging themselves.
My immediate question was about whether this closer to the current peak of performance, or just a lazy demonstration of OpenAI's power. So, I took one of the particularly interesting preprints to me (memory and precision in Gaussian models) and tested whether it's at the edge of capabilities or not. 30 minutes of back and forth with Astra (itself behind OAI's internal model) resulted in a significantly stronger result, on multiple dimensions. While I finished my first bottle, I spent a fair amount of time convincing myself of the result; I was convinced the strengthened results were plausible. Write this off as AI psychosis if you want, but try it yourself; I'm genuinely curious for what you get. (The back and forth, here, was entirely me saying "you can do it!", "keep at it, I believe in you", and "you've got this, finish it!")
Probably the most important line in OAI's announcement post is "The average result used the equivalent compute of roughly three hours of ChatGPT Pro thinking." This isn't a case of OAI spending millions for a marketing bump. My bet is a kind of Pareto distribution: most took minutes, not hours, with a long tail around Riemann-level results pulling up the average significantly.
Notably, ML related results are nearly entirely absent from this batch of proofs; maybe a half dozen touch on it, distantly. Some problem indices are skipped in overview.md. Conspiratorially, my inclination was to think they filtered them out for competitive advantage. I can't find any evidence of that in the GitHub repo or any of the preprints (equally plausible: deduping), so maybe they judiciously decided not to point their mathematical ballista at ML. ML also doesn't have a meaningful bank of rigorous conjectures, so given the conjecture sources, maybe they're not yet digging into ML math. But color me skeptical.
Does math matter? Is it something to advance civilization and technology, or an artistic pasttime for humans to create logical beauty? Likely both, today, but this is an almost nuclear detonation against the latter.
The only reason I'm not saying it's over is because I've said it before. It's been over for a while. And we've barely begun.
The good news, for mathematicians, is that their employability (in academia) did not hinge very strongly on tangible economic output. Nobody funds an algebraic geometer expecting a return on investment, which means nobody can defund one on the grounds that a datacenter in Texas now offers a better one.
The bad news? Everything else.
Just look at this fucking sweep. Just look at it. Hilbert's tenth over the rationals. Rational Hodge for CM abelian varieties. Integer multiplication below n log n, which I had mentally filed under "the floor, go home." A quasi-Riemann hypothesis thrown in like a free tote bag. Any one of these would have been the defining result of a human career, and they were released as a batch, in a GitHub repo, on a Tuesday, alongside hundreds of others.
And it wasn't brute force at absurd cost. OpenAI says the average result used roughly three hours of ChatGPT Pro-equivalent compute. I agree with your Pareto intuition, but the deeper point is that three hours is a price, and prices in this industry fall monotonically. Gwern spent 2020 arguing that neural nets would keep absorbing compute and sprouting new abilities, while noting the idea was so unpopular that it would only be accepted as a fait accompli. Well, here is the fait, and it is quite accompli. Next year the same theorem costs twenty minutes. The year after, it's a rounding error on someone's API bill.
Your own experiment deserves more attention than you gave it. As others have already done to good effect, your research methodology consisted of being a motivational poster ("you can do it!", "I believe in you") and you got a materially stronger result in half an hour. The binding constraint has shifted to elicitation and persistence.
Some might claim that human mathematicians are necessary to humanize and understand the proofs, or to come up with new frameworks and fields of research.
Maybe. I don't know. The strongest version of the argument is that mathematics is a conversation between humans about what humans find interesting, and a proof nobody understands is a tree falling in an empty forest. I'm sympathetic to that! I also notice that this position has been retreating to progressively smaller hills since the first computer-assisted proofs, and each hill gets defended with the same conviction as the last.
I do know that the current state of affairs is unlikely to last for more than six months anyway. Just get the models to come up with their own novel conjectures, or to attempt an overarching synthesis. I don't think turning abstruse Lean proofs into something human-readable will be particularly difficult, though human readability has long ceased to be a major concern for anyone. Note that OpenAI is shipping Lean formalizations for many of these proofs, so "trust me bro" is off the table; a type checker cares nothing for anyone's feelings. We're at the point where people are coping that the tsunami of new discoveries hinge on hither-to undiscovered bugs in Lean. You wish.
Every day, I feel the pace of progress accelerate, and then the practical ramifications show up in my life a few months later. I used to think of this as a lag. These days it feels more like the gap between seeing lightning and hearing thunder, and the gap keeps shrinking.
We've just had Scott Aaronson announce a new UT Austin course, CS395T: AI Alignment Theory, with Yudkowsky's AGI Ruin as the first assigned reading. To quote:
I'll give credit where it's due, but also: when the designated Reasonable Skeptic of the rationalist diaspora concedes the point in writing, the Overton window has relocated.
We've also just had an American company, Nolla Health, receive regulatory approval in Utah for AI to issue initial prescriptions. Yes, it's acne. Yes, the model can only choose from a short list of physician-approved topicals, and clinicians reportedly agreed with its recommendations in over 96% of cases. Everything starts as acne cream. Utah let Doctronic's AI handle prescription renewals back in January; nine months later, it's writing first-line scripts. You can extrapolate the line yourself. That's the harbinger of medicine's fall, as far as I'm concerned. I'm impressed we held the moat this long.
At least I've got a moat. The NHS is famously a slow ship to steer, and sclerotic at that. The future arrives unevenly distributed, and we can count on the service the last postcode on the delivery route. For once, I find that reassuring <3
Good career choice, past self_made_human. Turns out there are concrete benefits to taking theoretical but plausible concerns seriously, and preparing accordingly.
Sure, AI may be embarrassing humanity's proudest intellectual achievements. But there's a bright side. We can finally stop arguing about how AI is an overhyped bubble that's about to burst! Haven't even seen the term "stochastic parrot" in a while.
Another plus: AI can now compose its own Disney villain song. I'm not exactly sure where "writing a catchy diss track against people who think you're not intelligent" lies on the spectrum of intelligence, but it's probably a little ways past the Turing Test.
I'm sure we can do that, but maybe we shouldn't.
There's been some pretty not-great signs for the financial state of the industry lately: Microsoft looking to wean off of Anthropic, Harvey and Thompson Reuters moving to their own models, the force majeure notice by Oracle in New Mexico, the failure of SB Energy, Holtec, and Aggreko to IPO (not to mention OpenAI and Anthropic!), Firmus missing its rental payment, and generally increased skepticism on the part of investors.
I think it's been a huge problem for thinking clearly about the situation that the financial concern regarding the AI industry has been latched onto by the worst AI skeptics, who tend to flatly deny the capabilities of the models.
Meanwhile on the flip side, I suspect there may have been a parallel problem that the people who are the biggest boosters of the technology are the ones most likely to be suffering from mild AI psychosis from talking with them all the time.
Thus, the AI debate has been, somehow, between "AIs suck and there is a massive bubble" and "AIs are the best thing since sliced bread and nuh-uh," which excludes two entire quadrants of possibility from the conversation, "AIs suck and will be profitable" and "AIs are good and there is a bubble."
This state of discourse creates epistemic closure on the topic in a way that I think is preventing a lot of people from assessing where, exactly, AI is sitting.
The word "bubble" is hurting more than it's helping here. It's beyond question that the money spent on training frontier models will produce an absolutely massive amount of value for the world. But it's still to be seen how much of that value can be captured by the labs themselves. This isn't really the same shaped as tulip mania or people paying absurd amounts of money for pets.com. If the labs fail economically(Which I think could only happen if scaling laws collapse roughly tomorrow) then the weights will still be around and still worth quite a bit to serve on the hardware which it will still be around and be worth it to keep running inference on. The labs could fail but they wouldn't be failing because AI is overhyped or a scam, just that running a research company in a competitive environment without state granted monopolies on the produce of your research is a brutal business to be in.
It seems pretty plausible that the frontier labs could fail simply because demand doesn't meet their projections. Anthropic, for instance, has something like $200 billion in commitments to Google and Amazon out to 2036 where, according to the terms of the deal, they pay regardless of demand.
There's already signs that demand may soften for Anthropic specifically (the OpenRouter trendline is switching away from Anthropic towards open source models iirc, Microsoft is looking to cut their spend with them, ditto (we can infer) Harvey and Thompson Reuters, Astra is apparently universally beloved by coders). If people start switching from Anthropic (and it doesn't take many: keep in mind that 80% of Anthropic's revenue (like OpenAI's) comes from 1% of their users, and 2 of Anthropic's customers generated about 25% of their 2025 revenue), they can either
Except they can't cut a lot of their spend (as per above), if they raise prices, OpenAI eats them anyway, and perhaps Astra or OpenAI engineers are good enough that they are simply locked out of producing a superior product. That leaves #4, die.
Maybe this seems good for OpenAI, except that if the timing is bad it sends the market into an AI panic and could spoil their IPO, and OpenAI is also on the hook for a bunch of infrastructure bills (although they may have structured them more flexibly, I'm not sure offhand). And of course there are a lot of other things that could also ruin an IPO: another pandemic, major war breaking out in Europe or the Pacific or Middle East, political unrest: pretty much any little thing that goes wrong and tightens the belt could crack up the revenue stream, OpenAI is not a profitable company, and the open-source models are nipping at its heels.
Yes, I agree (and have said before) that "AI is not going anywhere." I agree this isn't the tulip mania. But we're in a subsidized era of AI right now, and it may look very different once that subsidy ends.
Perhaps the frontier labs fail for some other reason (or don't fail at all) but I think it's pretty fair to say that AI has been overhyped. OpenAI said it was going to spend $1.4 trillion on infrastructure by 2030. That's hyping. Then they slashed their public infrastructure commitments by more than half, to $600 billion (because their CFO was worried that they were overhyping), although they've brought the number back up since to $750 billion.
If they actually revise their numbers back up to $1.4 trillion in 2030 and meet that infrastructure goal, feel free to ping me and I will agree that this was a bad example. Then I will point you to the 2027 Project where it postulates that the robots would kill us all by now, as my fallback example.
This is basically describing what I mean by if scaling stops tomorrow. It can certainly be argued but it's my contention that if things keep scaling like this and all else holds steady the labs could just start directly eating segments of the economy. The "application layer" guys seem to expect the labs to just lay down and let them pay commodity prices to stand up a thin wrapper. But at a certain point anthropic can just instruct opus 7 "go ahead and stand up any product that would be profitable to run by a wrapper company, make no mistakes". That's a bit of a tongue in cheek example but what is stopping anthropic with a sufficiently advanced unreleased model from just becoming a hedge fund and doing to the financial industry what they're doing to mathematics? They have a wetlab, there's a lot of money in drug discovery.
I'm not making a strong claim here, I don't know what even the near future will bring. I don't think the labs failing is even all that unlikely. But I also don't think they'll have trouble finding investors money if they ask for it and it's not obvious that scaling laws have failed. I've personally set aside like a quarter of my net worth to be ready to invest in ant and oai at IPO.
I'll poke at the term subsidy just like the term bubble. Research is being subsidized, the inference is not. And it's hard to tell the degree of the research subsidy.
If you've been keeping score the 2027 project is doing much better than its detractors and pretty damn good in absolute terms.
Why haven't they done this already?
At least part of the answer to this question is probably that models often perform worse in real-world applications than their evaluation benchmarks would suggest. Thus, while the models are improving and becoming more useful for real world work, Anthropic can't just throw them at a problem and say "make no mistakes" without staffing up humans to oversee them.
Even in coding, which they have been optimized for, the data that I've seen indicates that trust by developers has fallen as adoption has risen, and that as LLMs have increasingly been adopted in coding, the amount of insecure code has risen substantially while deployments have actually slightly dropped.
Just as a personal example, the other day I asked Sonnet 5 to implement a design and it decided to simplify the product on its own recognizance. For me, it's a little annoying. But obviously at scale for a real-world product that's disastrous.
This is not an "LLMs are useless" post (I'm very very interested to see what comes out of the wetlab!), but it is a "LLMs are not ready to go to the moon on their own" post. It wouldn't surprise me if the primes try something like this, and they definitely might succeed. But it wouldn't be a "we can just let the AI do everything" sort of thing.
It seems like Anthropic and OpenAI think they will have at least momentary trouble, hence the delayed IPOs, and I noted the trouble the hardware suppliers were having upstream. If investors are becoming skeptical of datacenter investment, there's reason to think they are going to be skeptical of the primes. If they are skeptical of datacenter investment but not of the primes, they are just shooting themselves in the foot very dramatically, since the primes need the datacenters to scale. I suppose it's possible that they are all-in on orbital ones after the Space-X IPO, though.
This isn't a hard prediction here (nor do I think Anthropic and OpenAI delaying their IPOs is necessarily a bad idea or indicates terminal distress) but I think it's important to keep your eyes open. Particularly if you're intending to throw your net worth at them.
I'm getting inference for free, it's being subsidized.
The area where I think it did the worst was in the economic projection of the stock market as a whole, which I think is pretty relevant to the fundamental question here.
As I said, if the scaling stops tomorrow then maybe the models simply are not good enough to do this autonomously and other human labor is required to guide them along. If it continues at this rate for another year or two things change dramatically.
I don't have time right now to rehash this debate again. using sonnet instead of opus, not having a proper harness and planning/review cycle. I've seen the thing go, I've put out projects in weeks that would have taken months. If you don't believe it has the juice then so be it.
You're getting pennies of sonnet for free, I'm burning through $3k+ in tokens a month. Although the psychology here is confusing to me. You'd have to consider your time basically worthless to avoid playing $20/month for some opus 5.5 usage instead of sonnet 5.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link