This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
For years, the story of AI progress has been one of moving goalposts. First, it was chess. Deep Blue beat Kasparov in 1997, and people said, fine, chess is a well defined game of search and calculation, not true intelligence. Then it was Go, which has a state space so vast it requires "intuition." AlphaGo prevailed in 2016, and the skeptics said, alright, but these are still just board games with clear rules and win conditions. "True" intelligence is about ambiguity, creativity, and language. Then came the large language models, and the critique shifted again: they are just "stochastic parrots," excellent mimics who remix their training data without any real understanding. They can write a sonnet or a blog post, but they cannot perform multi step, abstract reasoning.
I present an existence proof:
OpenAI just claimed that a model of theirs qualifies for gold in the IMO:
To be clear, this isn't a production-ready model. It's going to be kept internal, because it's clearly unfinished. Looking at its output makes it obvious why that's the case, it's akin to hearing the muttering of a wild-haired maths professor as he's hacking away at a chalkboard. The aesthetics are easily excused, because the sums don't need one.
The more mathematically minded might enjoy going through the actual proofs. This unnamed model (which is not GPT-5) solved 5/6 of the problems correctly, under the same constraints as a human sitting the exam-
As much as AI skeptics and naysayers might wish otherwise, progress hasn't slowed. It certainly hasn't stalled outright. If a "stochastic parrot" is solving the IMO, I'm just going to shut up, and let it multiply on my behalf. If you're worse than a parrot, then have the good grace to feel ashamed about it.
The most potent argument against AI understanding has been its reliance on simple reward signals. In reinforcement learning for games, the reward is obvious: you won, or you lost. But how do you provide a reward signal for a multi page mathematical proof? The space of possible proofs is infinite, and most of them are wrong in subtle ways. Wei notes that their progress required moving beyond "the RL paradigm of clear cut, verifiable rewards."
How did they manage that? Do I look like I know? It's all secret-sauce. The recent breakthroughs in reasoning models like o1 and onwards relied heavily on "RLVR", which stands for reinforcement learning with verifiable reward. At its core, RLVR is a training method that refines AI models by giving them clear, objective feedback on their performance. Unlike Reinforcement Learning from Human Feedback (RLHF), which relies on subjective human preferences to guide the model, RLVR uses an automated "verifier" to tell the model whether its output is demonstrably correct. Presumably, Wei means something different here, instead of simply scaling up RLVR.
It's also important to note that previous SOTA, DeepMind's AlphaGeometry, a specialized system, had previously achieved a silver-medal level performance and was within spitting distance of gold. A significant milestone in its own right, but OpenAI's result comes from a general-purpose reasoning model. GPT-5 won't be as good at maths, either because it's being trained to be more general at the cost of sacrificing narrow capabilities, or because this model is too unwieldy to serve at a profit. I'll bet the farm on it being used to distill more mainstream models, and the most important fact is that it exists at all.
Update: To further show that isn't just a fluke, GDM also had a model that scored Gold this Olympiad. Unfortunately, in a very Google-like manner, they were stuck waiting for legal and marketing to sign off, and OAI beat them to the scoop.
https://x.com/ns123abc/status/1946631376385515829
https://x.com/zjasper666/status/1946650175063384091
"Moving the goalposts" is a bad metaphor.
Putting literal goal posts in approximately the right place is easy. Just put them on the end line, at the middle. But sports are competitive. Players will not be happy with the goal posts being in approximately the right place. They have to be in exactly the right place. This too is easy. Goal posts are self defining; the right place for the goal post is where the goal post is!
Belabouring the point, I invite you to consider a soccer match. In the first half, team A score with a shot just inside the right post. In the second half, team B fail to score with a shot just outside the left post. In the post-game adjudication, it is discovered that the goal posts were two feet right of they ought to be. Team A's goal gets disallowed. Team B's miss becomes a goal. Moving the goal posts flips a win for Team A into a win for Team B. The absurdity here is not so much the motion as the neglect. We are neglecting that the goal posts define the goal.
Turning now to Artificial Intelligence, we notice that humans are intelligent [citation needed :-)]. Which raises the question: why are we bothering to create an artificial version of what we already have? Mostly because the devil in in the details; humans are intelligent, but ...
If Alice copies Bob, and Bob copies Charles, and Charles copies Alice, then who should David copy? Human intelligence has a circle jerk problem. Perhaps David should copy Edward, who has reasoned things out from first principles. Perhaps David should copy Fiona who has done experiments. But brilliant, charismatic intectuals lead societies over cliffs. I wrote a paragraph on the difficulties of empirical science, but I deleted it because I couldn't get it to replicate.
We want something from Artificial Intelligence. We want it to cover the gaps in human intelligence. If we could crisply and accurately characterise those gaps we would be well on our way to fixing them ourselves. We have (had?) exactly one example of intelligence to look at, and we are not happy with it. We certainly notice that it has a weak meta-game: human intelligence is bad at seeing its own flaws. We are not able to install self-defining goal posts.
Old people bring baggage from the 1960's to discussion of AI. The word Computer invokes images of banks of tape drives reading databases. Human written legal briefs have a sloppiness problem. Need a precedent? A quick skim and this one looks close enough. It is job of the opposing lawyers to read it carefully and notice that it is not relevant. (The legal system is not supposed to work like this!) One images that an Artificial Intelligence actually reads the entire legal database and finds precedents that humans would miss. When an LLM invents a plausible, fictional precedent that just doesn't exist, one is taken by surprise. One wants to mark the AI down a lot for that non-human error. Doing so involves both moving the goal posts and admitting to not anticipating that failure mode at all.
There is a more subtle issue with LLMs writing computer programs. We may be underestimating the effort that goes into cleaning up LLM messes. LLMs learn to program from code bases written by humans. Not just written by humans, maintained by humans. So the bugs that humans spot and remove are under-represented in the training data. Meanwhile, the bugs that evade human skill at debugging lurk indefinitely and are over-represented in the training data. We have created tools to write code with bugs that humans have difficulty spotting. Worse, we estimate the quality of the code that our new tools produce on the basis that they are inhuman and have no special skill at writing bugs that we cannot spot, despite the nature of their training data.
Notice the clash with old-school expectations. A lot of GOFAI focussed on formal verification of mathematics. Some early theorem provers were ad hoc (and performed poorly). The attention shifted to algorithms growing out of Gödel's completeness theorem and Robinson's work on resolution theorem provers. Algorithms that were provably correct. The old school expectation involves a language such as SML, with a formal semantics, a methodology such as Dijkstra's "A Discipline of Programming", and code accompanied by a formally verified proof of correctness.
A tool for writing code with bugs that humans cannot find sounds like the kind of thing that Mossad would use to sabotage Iranian IT infrastructure. It may be super‐humanly intelligent, but we still want to move the goal posts to exclude it as the bad kind of intelligence.
One old school of AI imagined that the language of thought would be importantly different from natural language. The architecture of AI would involve translating natural language into a more rigorous and expressive internal language, thinking in this internal language and then translating back to natural language for output. LLMs do perhaps partially realise this dream. The tokens are placed in a multidimensional space and training involves discovering the latent structure, effectively inventing that training run's own, custom language of thought. If so, that is a win for the bitter lesson.
On the other hand, LLMs learn the world through human language. I believe that humans suffer from linguistic poverty. Many of our disputes bog down for lack of words. When we have one word for two concepts our discussions are reduced to hopping instead of walking.(My missing words web page is neglected, my post https://www.themotte.org/post/1043/splitting-defensive-alliance-into-chaining-alliance was not well liked, I'm not managing to explain the concept of linguistic poverty.) I hope that AI will "... cover the gaps in human intelligence." but LLMs seemed doomed to inherit our linguistic poverty and reproduce our existing confusions. The dream was that AI would cure human intellectual weakness not copy it.
I think that it is legitimate to notice that LLMs are indeed intelligent, and to then move the goal posts, declaring that, now we have seen it, we realise our error and this is not what we had in mind.
As I elaborated on in another comment in this thread, I do not think that some moving of goalposts is necessarily illegitimate. Our specifications can be incorrect, no one's immune from good old Goodhart.
Yet AI skeptics tend to make moving the goalposts into the entire sport. I will grant that their objections exist in a range of reasonableness, from genuine dissatisfaction with current approaches to AI, to Gary Marcus's not even wrong nonsense.
This is an interesting concern, and I mean that seriously. Fortunately, it doesn't seem to be empirically borne out. LLMs are increasingly better at solving all bugs, not just obvious-to-human ones. The ones in commercial production are not base models, naively concerned only with the next most likely token (and which necessarily includes subtle bugs that exist in the training distribution), but they're beaten into trying to find any and all bugs they can catch. Nothing in our (limited but not nonexistent) ability to interpret their behavior or cognition suggests that they're deliberately letting bugs through because they seem plausible. I am reasonably confident in making that claim, but I hope @faul_sname or @DaseindustriesLtd might chime in.
At the end of the day, there exist techniques like adversarial training to make such issues not a concern. Ideally, with formal verifications of code, you can't have unwanted behavior, ruled out by mathematical certainty. Of course, interpreting that you haven't made errors in formulating your specification is a challenge in itself.
There's been a decent amount of work done on dispensing with the need for tokenization in the first place, and letting the LLM operate/reason entirely in the latent space till it needs to output an answer. It seems to work, but hasn't been scaled to the same extent, and the benefits are debatable beyond perhaps solving minor tokenization errors that existing models have.
Human language, as used, is imprecise, but you can quite literally simulate a Turing machine with your speech. I don't see this as a major impediment, why can't LLMs come up with new words if needed, assuming there's a need for words at all?
I may or may not be an AI skeptic by your definition - I think it's quite likely that 2030 is a real year, and think it's plausible that even 2050 is a real year. But I think there genuinely is something missing from today's LLMs such that current LLMs generally fail to exhibit even the level of fluid intelligence exhibited by the average toddler (but can compensate to a surprising degree by leveraging encyclopedic knowledge).
My sneaking suspicion is that the "missing something" from today's LLMs is just "scale" - we're trying to match the capability of humans with 200M interconnected cortical microcolumns with transformers that only have 30k attention heads (not perfectly isomorphic, you could make the case that the correct analogy is microcolumn : attn head at a particular position, except the microcolumns can each have their own "weights" whereas the same attn head will have the same weights at every position), and we're trying to draw an equivalence between one LLM token and one human word. If you have an LLM agent that forks a new process in every situation in which a human would notice a new thing to track in the back of their mind, and allow each of those forked agents to define some test data and fine-tune / RL on it, I bet that'd look much more impressive (but also cost OOMs more than the current stuff you pay $200/mo for).
LLMs are increasingly better at solving a particular subset of bugs, which does not perfectly intersect the subset of bugs which humans are good at solving. Concretely, LLMs are much better at solving bugs that require them to know or shallowly infer some particular fact about the way a piece of code is supposed to be written, and fix it in an obvious way, and much much worse at solving bugs that require the solver to build up an internal model of what the code is supposed to be doing and an internal model of what the code actually does and spot (and fix) the difference. A particularly tough category of bug is "user reports this weird behavior" - the usual way a human would try to solve this is to try to figure out how to reproduce the issue in a controlled environment, and then to iteratively validate their expectations once they have figured out how to reproduce the bug. LLMs struggle at both the "figure out a repro case" step and the "iteratively validate assumptions" step.
In principle there is no reason LLMs can't come up with new words. There is precedence for the straight-up invention of language among groups of RL agents that start with no communication abilities and are incentivized to develop such abilities. So it's not some secret sauce that only humans have - but it is a secret sauce that LLMs don't seem to have all of yet.
LLMs do have some ingredients of the secret sauce: if you have some nebulous concept and you want to put a name to it, you can usually ask your LLM of choice and it will do a better job than 90% of professional humans who would be making that naming decision. Still, LLMs have a tendency not to actually coin new terms, and to fail to use the newly coined terms fluently in the rare cases that they do coin such a term (which is probably why they don't do it - if coining a new term was effective for problem solving, it would have been chiseled into their cognition by the RLVR process).
In terms of why this happens, Nostalgebraist has an excellent post on how LLMs process text, and how that processing is very different from how humans process text.
So there's a sense in which an LLM can coin a new term, but there's a sense in which it can't "practice" using that new term, and so can't really benefit from developing a cognitive shorthand. You can see the same thing with humans who try to learn all the jargon for a new field at once, before they've really grokked how it all fits together. I've seen it in programming, and I'm positive you've seen it in medicine.
BTW regarding the original point about LLM code introducing bugs - absolutely it does, the bugginess situation has gotten quite a bit worse as everyone tries to please investors by shoving AI this and AI that into every available workflow whether it makes sense to or not. We've developed tools to mitigate human fallibility, and we will develop tools to mitigate AI fallibility, so I am not particularly concerned with that problem over the long term.
Absolutely not, at least by standards! You acknowledge the possibility that we might get AGI in the near-term, and I see no firm reason to over index on a given year. Most people I'd call "skeptics" deny the possibility of AGI at all, or rule out any significant chance of near-term AGI, or have modal timelines >30 years.
I agree that LLMs are missing something, but I'm agnostic on whether brute-force scaling will get us to undisputable AGI. It may or may not. Perhaps online learning, as you hint at, might suffice.
I wonder if RLHF plays a role. I don't think human data annotators would be positively inclined towards models that made up novel words.
Thank you for taking the time to respond!
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link