site banner

Culture War Roundup for the week of August 4, 2025

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

3
Jump in the discussion.

No email address required.

A response to Freddie deBoer on AI hype

Bulverism is a waste of everyone's time

Freddie deBoer has a new edition of the article he writes about AI. Not, you’ll note, a new article about AI: my use of the definite article was quite intentional. For years, Freddie has been writing exactly one article about AI, repeating the same points he always makes more or less verbatim, repeatedly assuring his readers that nothing ever happens and there’s nothing to see here. Freddie’s AI article always consists of two discordant components inelegantly and incongruously kludged together:

  • sober-minded appeals to AI maximalists to temper their most breathless claims about the capabilities of this technology by carefully pointing out shortcomings therein

  • childish, juvenile insults directed at anyone who is even marginally more excited about the potential of this technology than he is, coupled with armchair psychoanalysis of the neuroses undergirding said excitement

What I find most frustrating about each repetition of Freddie’s AI article is that I agree with him on many of the particulars. While Nick Bostrom’s Superintelligence is, without exception, the most frightening book I’ve ever read in my life, and I do believe that our species will eventually invent artificial general intelligence — I nevertheless think the timeline for that event is quite a bit further out than the AI utopians and doomers would have us believe, and I think a lot of the hype around large language models (LLMs) in particular is unwarranted. And to lay my credentials on the table: I’m saying this as someone doesn’t work in the tech industry, who doesn’t have a backgrond in computer science, who hasn’t been following the developments in the AI space as closely as many have (presumably including Freddie), and who (contrary to the occasional accusation my commenters have fielded at me) has never used generative AI to compose text for this newsletter and never intends to.

I’m not here to take Freddie to task on his needlessly confrontational demeanour (something he rather hypocritically decries in his interlocutors), or attempt to put manners on him. If he can’t resist the temptation to pepper his well-articulated criticisms of reckless AI hypemongering with spiteful schoolyard zingers, that’s his business. But his article (just like every instance in the series preceding it) contains many examples of a particular species of fallacious reasoning I find incredibly irksome, regardless of the context in which it is used. I believe his arguments would have a vastly better reception among the AI maximalists he claims to want to persuade if he could only exercise a modicum of discipline and refrain from engaging in this specific category of argument.


Quick question: what’s the balance in your checking account?

If you’re a remotely sensible individual, it should be immediately obvious that there are a very limited number of ways in which you can find the information to answer this question accurately:

  1. Dropping into the nearest branch of your bank and asking them to confirm your balance (or phoning them).

  2. Logging into your bank account on your browser and checking the balance (or doing so via your banking app).

  3. Perhaps you did either #1 or #2 a few minutes before I asked the question, and can recite the balance from memory.

Now, supposing that you answered the question to the best of your knowledge, claiming that the balance of your checking account is, say, €2,000. Imagine that, in response, I rolled my eyes and scoffed that there’s no way your bank balance could possibly be €2,000, and the only reason that you’re claiming that that’s the real figure is because you’re embarrassed about your reckless spending habits. You would presumably retort that it’s very rude for me to accuse you of lying, that you were accurately reciting your bank balance to the best of your knowledge, and furthermore how dare I suggest that you’re bad with money when in fact you’re one of the most fiscally responsible people in your entire social circle—

Wait. Stop. Can you see what a tremendous waste of time this line of discussion is for both of us?

Either your bank balance is €2,000, or it isn’t. The only ways to find out what it is are the three methods outlined above. If I have good reason to believe that the claimed figure is inaccurate (say, because I was looking over your shoulder when you were checking your banking app; or because you recently claimed to be short of money and asked me for financial assistance), then I should come out and argue that. But as amusing as it might be for me to practise armchair psychoanalysis about how the only reason you’re claiming that the balance is €2,000 is because of this or that complex or neurosis, it won’t bring me one iota closer to finding out what the real figure is. It accomplishes nothing.

This particular species of fallacious argument is called Bulverism, and refers to any instance in which, rather than debating the truth or falsity of a specific claim, an interlocutor assumes that the claim is false and expounds on the underlying motivations of the person who advanced it. The checking accout balance example above is not original to me, but from C.S. Lewis, who coined the term:

You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly.

As Lewis notes, if I have definitively demonstrated that the claim is wrong — that there’s no possible way your bank balance really is €2,000 — it may be of interest to consider the psychological factors that resulted in you claiming otherwise. Maybe you really were lying to me because you’re embarrassed about your fiscal irresponsibility; maybe you were mistakenly looking at the balance of your savings account rather than your checking account; maybe you have undiagnosed myopia and you misread a 3 as a 2. But until I’ve established that you are wrong, it’s a colossal waste of my time and yours to expound at length on the state of mind that led you to erroneously conclude that the balance is €2,000 when it’s really something else.

In the eight decades since Lewis coined the term, the popularity of this fallacious argumentative strategy shows no signs of abating, and is routinely employed by people at every point on the political spectrum against everyone else. You’ll have evolutionists claiming that the only reason people endorse young-Earth creationism is because the idea of humans evolving from animals makes them uncomfortable; creationists claiming that the only reason evolutionists endorse evolution is because they’ve fallen for the epistemic trap of Scientism™ and can’t accept that not everything can be deduced from observation alone; climate-change deniers claiming that the only reason environmentalists claim that climate change is happening is because they want to instate global communism; environmentalists claiming that the only reason people deny that climate change is happening is because they’re shills for petrochemical companies. And of course, identity politics of all stripes (in particular standpoint epistemology and other ways of knowing) is Bulverism with a V8 engine: is there any debate strategy less productive than “you’re only saying that because you’re a privileged cishet white male”? It’s all wonderfully amusing — what could be more fun than confecting psychological just-so stories about your ideological opponents in order to insult them with a thin veneer of cod-academic therapyspeak?

But it’s also, ultimately, a waste of time. The only way to find out the balance of your checking account is to check the balance on your checking account — idle speculation on the psychological factors that caused you to claim that the balance was X when it was really Y are futile until it has been established that it really is Y rather than X. And so it goes with all claims of truth or falsity. Hypothetically, it could be literally true that 100% of the people who endorse evolution have fallen for the epistemic trap of Scientism™ and so on and so forth. Even if that was the case, that wouldn’t tell us a thing about whether evolution is literally true.


To give Freddie credit where it’s due, the various iterations of his AI article do not consist solely of him assuming that AI maximalists are wrong and speculating on the psychological factors that caused them to be so. He does attempt, with no small amount of rigour, to demonstrate that they are wrong on the facts: pointing out major shortcomings in the current state of the LLM art; citing specific examples of AI predictions which conspicuously failed to come to pass; comparing the recent impact of LLMs on human society with other hugely influential technologies (electricity, indoor plumbing, antibiotics etc.) in order to make the case that LLMs have been nowhere near as influential on our society as the maximalists would like to believe. This is what a sensible debate about the merits of LLMs and projections about their future capabilities should look like.

But poor Freddie just can’t help himself, so in addition to all of this sensible sober-minded analysis, he insists on wasting his readers’ time with endless interminable paragraphs of armchair psychoanalysis about how the AI maximalists came to arrive at their deluded worldviews:

What [Scott] Alexander and [Yascha] Mounk are saying, what the endlessly enraged throngs on LessWrong and Reddit are saying, ultimately what Thompson and Klein and Roose and Newton and so many others are saying in more sober tones, is not really about AI at all. Their line on all of this isn’t about technology, if you can follow it to the root. They’re saying, instead, take this weight from off of me. Let me live in a different world than this one. Set me free, free from this mundane life of pointless meetings, student loan payments, commuting home through the traffic, remembering to cancel that one streaming service after you finish watching a show, email unsubscribe buttons that don’t work, your cousin sending you hustle culture memes, gritty coffee, forced updates to your phone’s software that make it slower for no discernible benefit, trying and failing to get concert tickets, trying to come up with zingers to impress your coworkers on Slack…. And, you know, disease, aging, infirmity, death.

Am I disagreeing with any of the above? Not at all: whenever anyone is making breathless claims about the potential near-future impacts of some new technology, I have to assume there’s some amount of wishful thinking or motivated reasoning at play.

No: what I’m saying to Freddie is that his analysis, even if true, doesn’t fucking matter. It’s irrelevant. It could well be the case that 100% of the AI maximalists are only breathlessly touting the immediate future of AI on human society because they’re too scared to confront the reality of a world characterised by boredom, drudgery, infirmity and mortality. But even if that was the case, that wouldn’t tell us one single solitary thing about whether this or that AI prediction is likely to come to pass or not. The only way to answer that question to our satisfaction is to soberly and dispassionately look at the state of the evidence, the facts on the ground, resisting the temptation to get caught up in hype or reflexive dismissal. If it ultimately turns out that LLMs are a blind alley, there will be plenty of time to gloat about the psychological factors that caused the AI maximalists to believe otherwise. Doing so before it has been conclusively shown that LLMs are a blind alley is a waste of words.

Freddie, I plead with you: stay on topic. I’m sure it feels good to call everyone who’s more excited than you about AI an emotionally stunted manchild afraid to confront the real world, but it’s not a productive contribution to the debate. Resist the temptation to psychoanalyse people you disagree with, something you’ve complained about people doing to you (in the form of suggesting that your latest article is so off the wall that it could only be the product of a manic episode) on many occasions. The only way to check the balance of someone’s checking account is to check the balance on their checking account. Anything else is a waste of everyone’s time.

In the eight decades since Lewis coined the term, the popularity of this fallacious argumentative strategy shows no signs of abating, and is routinely employed by people at every point on the political spectrum against everyone else. You’ll have evolutionists claiming that the only reason people endorse young-Earth creationism is because the idea of humans evolving from animals makes them uncomfortable; creationists claiming that the only reason evolutionists endorse evolution is because they’ve fallen for the epistemic trap of Scientism™ and can’t accept that not everything can be deduced from observation alone; climate-change deniers claiming that the only reason environmentalists claim that climate change is happening is because they want to instate global communism; environmentalists claiming that the only reason people deny that climate change is happening is because they’re shills for petrochemical companies. And of course, identity politics of all stripes (in particular standpoint epistemology and other ways of knowing) is Bulverism with a V8 engine: is there any debate strategy less productive than “you’re only saying that because you’re a privileged cishet white male”? It’s all wonderfully amusing — what could be more fun than confecting psychological just-so stories about your ideological opponents in order to insult them with a thin veneer of cod-academic therapyspeak?

Your post is overall good, but I think you take this part too far. There are questions, indeed including on issues you've listed here, where a genuine issue of material fact exists, and is not and likely cannot be resolved in the near term.

My example would be climate change. I have slight confidence, approximately 65%ish that the climate is warming faster than it would without human CO2 emissions. This is hardly the sort of confidence level one should have if you are deciding major issues. It gets even lower when I ask the question, "assuming it is true the climate is warming because of human CO2 emissions, is that bad?" On even that question we are at 50% max, most credible people I have looked at seem to indicate slight warming is probably good for the earth and humanity. And then there is the next question of, will the policy proposed by this politician/advocate meaningfully change the outcome, and there I estimate abysmal results in the 1-5% range.

So I am left with a confidence chart of(when being favorable to environmentalists): A) Global Warming is true and humans contribute: 70% B) That is bad: 50% C) The proposed policies can fix it: 5%

For a composite confidence level of 1.75% that environmentalist proposals will solve the problem they are purporting to solve.

And yet, environmentalists act as if they have 100% confidence, and they commonly reject market solutions in favor of central planning. The logical deduction from this pattern of behavior is that the central planning is the goal, and the global warming is the excuse. It is not bad argumentation to say to the environmentalist, "you are just a socialist that wants to control the economy, and are using CO2 as an excuse" because a principled environmentalist would never bother raising a finger in America. They'd go to India and chain themselves to a river barge dumping plastic or go to Africa and spay and neuter humans over there. If you are trying to mess with American's cars, heat, and AC, its because you dont like that Americans have those things, because other concerns regarding the environment have been much more pressing for several decades at this point, and that isn't likely to change.

And yet, environmentalists act as if they have 100% confidence, and they commonly reject market solutions in favor of central planning. The logical deduction from this pattern of behavior is that the central planning is the goal, and the global warming is the excuse. It is not bad argumentation to say to the environmentalist, "you are just a socialist that wants to control the economy, and are using CO2 as an excuse" because a principled environmentalist would never bother raising a finger in America. They'd go to India and chain themselves to a river barge dumping plastic or go to Africa and spay and neuter humans over there. If you are trying to mess with American's cars, heat, and AC, its because you dont like that Americans have those things, because other concerns regarding the environment have been much more pressing for several decades at this point, and that isn't likely to change.

This is a failure of theory of mind.

As a general rule, when there's a situation where person A insistently tries to solve problem B with method C rather than more-effective method D, the conclusion "A is secretly a liar about wanting to solve problem B and just wants to do method C for other reason E" is almost always false, outside of special cases like PR departments and to some extent politicians. The correct conclusion is more often "A is not a consequentialist and considers method D sinful and thus off the table". "A thinks method D is actually not more effective than method C" is also a thing.

So, yes, a lot of these people really are socialists, but they're also environmentalists who sincerely believe CO2 might cause TEOTWAWKI. It's just, well, you actually also need the premise of "sometimes there isn't a perfect solution; pick the lesser evil" in order to get to "pursue this within capitalism rather than demanding we dismantle capitalism at the same time", and a lot of people don't believe that premise.

One thing I often find annoying about this sort of conversation is we are often talking about someone's internal state of mind. I find that irrelevant. See also trans/gay and "groomer." When your outward actions are indistinguishable from the person that thinks the thing I think they think, they have to bring the chits. This is true whether you are a teacher talking to 13 year olds in private about how anal is great and they don't need to share this conversation with their parents, or whether you are a legislator voting for an arcane and complex regulatory system that will commandeer 1/10th of the economy instead of a simple flat tax.

Edit: And by the way, the suspicion gets worse because you, Mr. Legislator, always want the arcane and complex regulatory system whether it is carbon, or medicine, or banking... Very suspicious.

What's a "chit"? I think that's an Americanism.

I would say that your assumptions are way off with regards to your opponents.

The environmentalist would say that:

A) Global Warming is true and humans contribute: 99%

B) That is bad: 95% (addendum to B) How bad? Coastal areas become flooded. Increased deaths due to heat. Increased frequency of natural disasters. Risk to food production due to unstable weather. We're talking potentially millions of deaths.

C) That due to the severity estimates of B, if proposed policies won't work, increased attention must be given to come up with policies that will.

And yet, environmentalists act as if they have 100% confidence, and they commonly reject market solutions in favor of central planning. The logical deduction from this pattern of behavior is that the central planning is the goal, and the global warming is the excuse.

Alternatively, their confidence levels of capitalism solving societal problems are low as a general rule, and in this particular instance capitalism literally has negative incentives to solving this problem. How does one monetize the ability to bring the global temperature of Earth down? How do you monetize clean air? The only thing I can think of are carbon credits, and Republicans oppose implementing such a system in the first place. Conversely, you make money by producing things which pollute the air as a byproduct, and putting in effort to mitigate pollution costs money for no monetary benefit.

a principled environmentalist would never bother raising a finger in America. They'd go to India and chain themselves to a river barge dumping plastic or go to Africa and spay and neuter humans over there.

This strikes me as comparable to saying that pro-life people should be shooting up abortion clinics. Your average environmentalist believes they have 0 influence on India's policies because they are not Indian, and India's government gives every impression they don't care. People believe lots of things they don't follow through fully on, even assuming the assertions that if they believe X they should do Y are reasonable. Do all the people complaining about western fertility have 10 kids?

If you are trying to mess with American's cars, heat, and AC, its because you dont like that Americans have those things, because other concerns regarding the environment have been much more pressing for several decades at this point, and that isn't likely to change.

That's incredibly boo outgroup. How many people out there actually hate that people have things, as a primary motivation?

A) Global Warming is true and humans contribute: 99%

B) That is bad: 95% (addendum to B) How bad? Coastal areas become flooded. Increased deaths due to heat. Increased frequency of natural disasters. Risk to food production due to unstable weather. We're talking potentially millions of deaths.

C) That due to the severity estimates of B, if proposed policies won't work, increased attention must be given to come up with policies that will.

Those number are just bonkers, and makes reasonable people want to just write off environmentalists (if true).

Alternatively, their confidence levels of capitalism solving societal problems are low as a general rule, and in this particular instance capitalism literally has negative incentives to solving this problem. How does one monetize the ability to bring the global temperature of Earth down? How do you monetize clean air? The only thing I can think of are carbon credits, and Republicans oppose implementing such a system in the first place. Conversely, you make money by producing things which pollute the air as a byproduct, and putting in effort to mitigate pollution costs money for no monetary benefit.

But despite the failure of environmentalists to implement their preferred policies, capitalism has brought down carbon emissions in the US quite a bit. So this part is just them screaming that reality is wrong. And actually, its easy to use the market to monetize reducing carbon usage. You put a carbon tax at a per ton basis and then you cut taxes elsewhere to make sure that the carbon tax doesn't cripple the economy. Notably when such a proposal was made in Washington State, environmentalists were part of the coalition that killed the proposal. The fact that they have this specific bundle of beliefs appears problematic for your thesis. The central planning thesis is actually strengthened here.

This strikes me as comparable to saying that pro-life people should be shooting up abortion clinics. Your average environmentalist believes they have 0 influence on India's policies because they are not Indian, and India's government gives every impression they don't care. People believe lots of things they don't follow through fully on, even assuming the assertions that if they believe X they should do Y are reasonable. Do all the people complaining about western fertility have 10 kids?

Valid to an extent. But, what if this is just another irrational part of their bundle of beliefs. That being that being an environmentalist and anti-capitalist is also highly correlated with being...anti-white/western? Perhaps all these beliefs are in conflict for achieving each seperate stated goal, except as to the part where all the policies trend toward...more central planning.

That's incredibly boo outgroup. How many people out there actually hate that people have things, as a primary motivation?

Lots? There are a bunch of large subreddits dedicated to these beliefs like /r/fuckcars, and the mods of those are typically powermods that also control super large subs like /r/politics and /r/askreddit.

Those number are just bonkers, and makes reasonable people want to just write off environmentalists (if true).

I was personally just making up high numbers, but over the long term (meaning I make no prediction about if it will be 5 years from now or 500), I do believe these things to be true.

But despite the failure of environmentalists to implement their preferred policies, capitalism has brought down carbon emissions in the US quite a bit. So this part is just them screaming that reality is wrong. And actually, its easy to use the market to monetize reducing carbon usage. You put a carbon tax at a per ton basis and then you cut taxes elsewhere to make sure that the carbon tax doesn't cripple the economy. Notably when such a proposal was made in Washington State, environmentalists were part of the coalition that killed the proposal. The fact that they have this specific bundle of beliefs appears problematic for your thesis. The central planning thesis is actually strengthened here.

That's easy to square. Capitalism created the pollution, particularly during the industrial revolution when pollution was largely ignored, then government (not capitalism) intervened to force companies to change. Having government push companies to reduce pollution is their preferred policy and was enacted, if not to the extent that they want.

Reading your link, it sounds to me like they believed that if they killed this bill they could get a new, more aggressive version pushed. Progressives letting the perfect be the enemy of the good is nothing new.

Valid to an extent. But, what if this is just another irrational part of their bundle of beliefs. That being that being an environmentalist and anti-capitalist is also highly correlated with being...anti-white/western? Perhaps all these beliefs are in conflict for achieving each seperate stated goal, except as to the part where all the policies trend toward...more central planning.

The reason I'm a Democrat but not a progressive is because I think that progressives are somewhat good at identifying problems (if oversensitive) but bad at solving them. It's the same personality trait that lead to becoming an environmentalist that lead to every other cause du jour.

I don't think brainstorming solutions to problems is bad, I just think they tend to weight real-life problems high and problems with their hypothetical alternatives low. They aren't central planning for the sake of central planning, they're central planning because it is the most obvious instrument that could potentially do all the things they feel must be done.

Lots? There are a bunch of large subreddits dedicated to these beliefs like /r/fuckcars, and the mods of those are typically powermods that also control super large subs like /r/politics and /r/askreddit.

True, I was not thinking of fuckcars. I think I'd only really heard the name once. A quick scan seems to me that their primary issue with cars is the number of people who die in car accidents. I disagree, but that does sound like a motivation that cars are harmful rather than a motivation that because they don't like cars that nobody should have them. Though to be fair I am also seeing some who do hate cars, mostly due to hating parking lots.

That's incredibly boo outgroup. How many people out there actually hate that people have things, as a primary motivation?

The hairshirt environmentalist is not at all uncommon. Just because something is uncomplimentary to the outgroup does not mean it is not true.

I live in a college town. I honestly can't think of a single person right now in real life that I would describe as a hairshirt environmentalist. Online, I can only think of Greta herself and her refusal to take an airplane, and she's a massive outlier because she pretty much uses her influence to bum rides around the world on an eco-friendly yacht. A quick check of Just Stop Oil shows that most of their antics result in 50-ish arrests, which seems like peanuts to me.

Your average environmentalist is a middle class college kid with an iPhone. They aren't giving up much of anything except maybe biking more and eating less meat.

With the caveat Rasmussen and N=1000.

Your average environmentalist is a middle class college kid with an iPhone. They aren't giving up much of anything except maybe biking more and eating less meat.

Perhaps not, but they want YOU to give up your car, your air travel, your air conditioning, your single-family home and yard, your meat, etc.

But how much do they want that?

Saying “I think St. Thomas was pretty admirable” isn’t the same as putting on the hairshirt yourself.

But how much do they want that?

Enough to support politicians and policies which will result in people being forced to do it.

Your average environmentalist is a middle class college kid with an iPhone. They aren't giving up much of anything except maybe biking more and eating less meat.

The comment I've heard several times from middle class environmentalist friends is, "Of course, people are going to have to stop doing [thing I don't do]". Biking and recycling make them feel that they've made their sacrifices and they can happily start requiring things from other people.

Oh, I absolutely agree that their actions are often superficial and having unreasonable expectations of others. That was part of my backhand comment about college kids with iPhones. It's much like wanting to lose weight but not dieting (outside of switching to diet soda) or exercising.

My point of disagreement was anti-dan's framing was that they're not actually motivated by a desire to reduce pollution, instead they want people to live worse lives for the hell of it I guess? Because they derive enjoyment out of decreasing the total happiness in the world or something?

My model is that lots of people want to have their cake and eat it too. That they end up eating the cake is because obviously they can't have both and base desires won out. I'm more objecting to what I see as someone going:

I keep seeing people saying you should save your cake but they end up eating it. The logical conclusion is they hate the sight of cake and want to destroy it.

I think there's some of both. Someone was talking last week about how much environmentalism is an aesthetic: happy, multi-coloured people in harmony with nature and each other, living in beautiful garden cities. And that aesthetic is both positive and negative to some degree. Pro-local neighbourhoods has to mean anti-car, pro-clean-air means anti-smoke and therefore anti-factory, anti-wood-fires, anti-gas-hobs etc.

I think @anti-dan is correct in that often the 'anti-' aesthetic comes first, people dislike chaos and capitalism and want central planning, they dislike 'dirty' industry, they dislike racism and nationalism and parocialism and this plays a big role in their willingness to become Greens and to believe the more extreme takes on that side.

As always, I default to Bertrand Russel's method: any deeply held belief requires at least two of [personal desire, +/- social pressure, and preponderance of empirical evidence]. You will believe something if you really like it and the evidence seems to line up that way (HBD, often), or if you like it and your community agrees even though the evidence doesn't really line up that way (most religion inc. mine IMHO as a Christian), or if the evidence lines up that way and there is social consensus (we're probably not going to get lots out of interstellar space races).

The Puritan impulse that is driven by "The haunting fear that someone, somewhere, may be happy" is not limited to guys in funny hats and capes

In point of fact, I do literally believe that a great many Western environmentalists are only tooting the horn about climate change as a convenient pretext to instate global communism or something approximating it. (I think Greta Thunberg had a bit of a mask-off moment in which she more or less copped to this.) But even if that was true of 100% of them, it wouldn't change the factual question of whether or not the earth is actually getting hotter because of human activity. "You're only sounding the alarm as a pretext to instate global communism" could be literally true of the entire movement's motivations, and yet completely irrelevant for the narrow question of fact under discussion.

many Western environmentalists are only tooting the horn about climate change as a convenient pretext to instate global communism

I started drafting a top level post last week touching on a related trend but I didn't have enough to round it out. I still don't really, but I think there's something in there. Anyway, last week I saw a poster for an Alternative Pride March in my city. My CW radar was pinged, I looked it up and found out that it's explicitly Marxist ("Pride without cops or corporations!" etc). Events under the same banner are being organised in other cities suggesting it's unlikely to be a grass roots movement.

So now the social acceptance and establishment endorsement of LGBT is... bad? That doesn't seem convincing.

What I think is happening is that Marxists prey on these fringe movements. It's not that LGBT and environmentalists are eager for Communism. I think many of them are sincere that what they want is no more than reasonable policies addressing their defined political interest (gay marriage, say, or clean rivers). I suspect that Marxists court and enter these movements that are made of what are already soft radicals who are acculturated to being unhappy with an aspect of the status quo and begin efforts to turn them into hard radicals who will become convinced that the status quo has to be disposed of wholesale (our revolution is necessary for the sake of your own cause, Comrade).

That is to say it's not environmentalists who are tooting the climate change horn to instate Communism, nor is it LGBTs, it's Communists. The fact that these movements are already socially accepted gives them cover to expand the agenda because now they can condemn resistance to the veiled Communist ideology as eg ecocidal transphobia.

But even if that was true of 100% of them, it wouldn't change the factual question of whether or not the earth is actually getting hotter because of human activity.

It wouldn't change the question but it would change the most credible answer (if you didn't already believe it, anyway). Ad hominem is a formal fallacy, but once you're out of the purely formal world, the credibility of the people making the propositions matter. If everyone sounding the alarm is doing it as a pretext to instate global communism, chances are good it's a false alarm.

I too, found this article extremely annoying. This guy is for real accusing, Scott Alexander of all people, of not laying out his opinions and justifications of ai acceleration in enough detail? Could he have maybe tried reading any of how his writing on the topic?

This is basically all of FdB's articles lately. Lots of words to complain about how people care too much about things he doesn't care about, and like things he doesn't like. He's lost his edge worse than Scott.

He has mentioned declining subscriptions and revenue. I wonder if he's re-running a few of his 'greatest hits' to see what still strikes a chord?

He may have suffered the sad fate of many pundits, though - he's run out of things to say. He can either re-run older points he's made in increasingly angry or provocative ways, as he is here, or he can pivot to increasingly niche content, as I think you can see in some of his other recent essays.

With that paragraph you cite, I wonder how Freddie would feel if one swapped out "AI" with "Marxism."

I know, it's weird for a self-proclaimed Marxist to criticise others for being unrealistically utopian.

That's all true, but the best argument against Freddie deBoer isn't a bunch of words but just to point out that he suffers from severe mental illness and is desperately trying to shed that reputation he's gained by attacking overly enthusiastic nerds.

What is the relationship between attacking enthusiastic nerds specifically and trying to shed a reputation for mental illness?

This isn't a great comment, from a moderation perspective, but I'm now going to talk as a general user.

Freddie is seriously mentally ill. He's disclosed his struggle with bipolar disorder (and I seem to recall he had psychotic breaks). However, the views he espoused aren't exclusive to the mentally ill. God knows I would like to pin a diagnosis on someone like Gary Marcus, but being pugnacious and immune to evidence isn't actually in the DSM.

As far as I can tell, he does have 'aesthetic' reasons to object to transhumanist/rationalist viewpoints. He is woefully under-informed about modern AI, but so are most people. It is easy to demolish his arguments without needing to resort to ad hominem attacks. If he starts accusing OAI of wire-tapping his house (like Altman's sister), then this facet of his personality would be worth raising.

Makes one wonder about the definition of mental illness. The European generals pre WW1 who still kept lancer cavalry regiments might be considered 'retarded' by an observer, even without the benefit of hindsight. WTF are lancers gonna do to bolt action riflemen, let alone machineguns? Even if you just imagine the infantry out there on a field, unentrenched and in loose order, in the best conditions for a cavalry charge, the whole thing is still a slaughter.

But this kind of 'retarded', distinct from being an actual dribbler who probably has no concept of what lancers are, isn't an actual mental illness, it's just being really really bad at your job in one specific area. Kind of like an AI hallucination in scope, an isolated lapse in otherwise reasonable performance.

But instead of a random hallucination it's more like a motivated argument where one might despise the grifters on twitter or dislike the cut of Yudkowsky's jib and then form your opinions based on that. Likewise, I imagine the cavalry officers held themselves high, cavalry is noble and aristocratic, a testament to the connection between man and horse. Infantry was ugly muddy and plebeian, so they looked for reasons why the cavalry should win when the idea is idiotic.

WTF are lancers gonna do to bolt action riflemen, let alone machineguns?

Cavalry did pretty well in the ACW, and in the Franco-Prussian war.

I find the tactical insanity of WW1 pretty understandable if you remember that bolt action rifles and light machine guns are incremental changes. It was hard to foresee that just making everything slightly faster and more portable would make most doctrine obsolete.

There was a tendency for cavalry to get lighter and serve more as scouts than shock forces. But the total obsolescence of the concept was hard to fathom .

Moreover, outside of the Western front, cavalry did an outstanding job in WW1 even. Both on the eastern front and the Balkans with fast moving fronts, the advantages of mobility start to outweigh firepower.

It's only in WW2 with the infamous polish failures that cavalry was rendered soundly obsolete. And only really because motorized units took over the role.

It's far more understandable to me than some air forces deciding to stick to scouting and refusing to entertain combat flight despite obvious trends. But then again, the future of aviation was as mysterious as that of AI today at the time.

It's only in WW2 with the infamous polish failures that cavalry was rendered soundly obsolete.

I just want to note that these failures were partially fiction invented by German propaganda.

(also, Wermach being fully mechanized is also a myth, they used far more horses than people know)

IIRC there were still literally millions of horses used by both the Nazis and the Soviets in WWII, and the Nazi invasion of the USSR used more horses than trucks.

But I was under the impression that these uses were more for logistics than for combat, which would make a lot more sense to me. "We're low on oil; we shouldn't forgo transports whose fuel literally grows right out of the ground", yes. "Into the mouth of Hell, Rode the six hundred, But this time they had, A Good feeling about it", no, right?

Or maybe I'm completely wrong. The Nazis took Poland in 5 weeks, so I don't want to give the defense too much credit, but digging into the details it really looks like "using horse-mounted cavalry, even against mechanized infantry" wasn't a problem, it just wasn't good enough to overcome "being simultaneously invaded by two foes with at least twice their population each".

But I was under the impression that these uses were more for logistics than for combat

Genuinely unsure, but combat use was a thing, see for example https://en.wikipedia.org/wiki/1st_Cavalry_Division_(Wehrmacht)

It was withdrawn to France in November 1941 and its 17,000 horses were handed over to infantry divisions.


so I don't want to give the defense too much credit

definitely, it was overall poor showing in many ways, but admittedly "being simultaneously invaded by two foes with at least twice their population each" was hard to overcome

And it was not at "planning to charge tanks with cavalry" levels, cavalry against tanks was closer to "use anti-tank guns, relocate, repeat" or "cut down infantry with mobile cavalry while hilariously poorly mobile early tanks are uselessly stuck somewhere else". And if it would be Germany vs Poland + maybe France etc there was some chance.

Sure there was still a role for cavalry as mounted dragoons or scouts in WW1 and WW2 but real European doctrine was theorizing actual cavalry charges with lances and sabers.

I'm pretty sure the real European doctrine for cavalry units in WW2 was to use them as mounted infantry units that dismount and shoot guns during combat.

(The infamous failures of Polish cavalry charging at tanks supposedly didn't actually happen, though they did charge at some infantry formations a couple of times, to mixed results.)

To be more specific, IIRC aftermath of what was used in German propaganda was case of Polish cavalry demolishing German infantry with an actual charge and in turn being demolished by German tanks that arrived later.

BTW, Germany also had frontline cavalry units in WW II.

WW1 on Eastern front and 1920 war between Poland and Russia had actual cavalry charges with lances and sabers (and guns).

(in the 1920 it worked so well that it contributed both sides to overvaluing cavalry, though neither was planning* to charge tanks with them by the time of WW II)

*or actually charging

Because that did still work in the previous wars where automatic weapons were bulky crew served emplacements. During the Franco-Prussian war heavy cavalry still did its intended saber charge role at times using smoke cover and violence of action to make movement and concealment nullify firepower.

The thinking was that the trend would continue with minor adjustments.

A potential analog today is the ever prophetized death of the tank and armored offensives. In the face of top easily available countermeasures. People said the prevalence of ATGMs would sunset the tank and it did not, so now we expect that the prevalence of drones won't, but maybe we are all insane and the future will regard armored offensives by Russia and Ukraine as doomed follies.

My point is, it's hard to judge the past decision makers honestly without tainting it with our knowledge of the outcome.

And sometimes what becomes the conventional wisdom also goes too far. Bayonet charges have in fact decided some battles in the Falklands despite expectations for instance. Despite both belligerents having access to vastly more sophisticated weapons than spears.

ATGMs have countermeasures, you can have active defences or redesign armour to resist them better. Drones have countermeasures, you can cover the tank in add-on armour like we see in Ukraine. Or redesign the tank to be more well-rounded in its armour rather than so frontally-focused. You can add ECM, some microwave widget, have defensive drones.

But you can't redesign heavy shock cavalry in the same way. You can add more armour but the horse biology and blade technology hasn't advanced significantly for ages.

During the Franco-Prussian war cavalry charges did occasionally work but at great cost. Since there were no further advances in cavalry but great advancements in rifles, artillery and machine-guns (and accompanying tactics, indirect fire and entrenchment) then traditional shock cavalry was foreseeably obsolete.

Likewise, if drones turn into autonomous AI death swarms with tandem warheads, doubled range and halved price while tanks remain fundamentally in the 1980s, then it would be all over for heavy armour. But that won't necessarily happen since we know the tank has all these opportunities to adapt that cavalry lack.

Chat, is this Bulverism?

the best argument against Freddie deBoer isn't a bunch of words but just to point out that he suffers from severe mental illness and is desperately trying to shed that reputation

Chat, is this Bulverism?

Textbook case.

No: what I’m saying to Freddie is that his analysis, even if true, doesn’t fucking matter. It’s irrelevant. It could well be the case that 100% of the AI maximalists are only breathlessly touting the immediate future of AI on human society because they’re too scared to confront the reality of a world characterised by boredom, drudgery, infirmity and mortality. But even if that was the case, that wouldn’t tell us one single solitary thing about whether this or that AI prediction is likely to come to pass or not. The only way to answer that question to our satisfaction is to soberly and dispassionately look at the state of the evidence, the facts on the ground, resisting the temptation to get caught up in hype or reflexive dismissal. If it ultimately turns out that LLMs are a blind alley, there will be plenty of time to gloat about the psychological factors that caused the AI maximalists to believe otherwise. Doing so before it has been conclusively shown that LLMs are a blind alley is a waste of words.

Disagree. If it was true, it would matter quite a bit.

If deBoer was right- both in his conclusion and his reasoning as to why- it would be really relevant. It would mean, among other things, that deBoer had an actually, insightful, accurate, and predictive model of notoriously difficult fields of technology and human pyschology that can all be used to know results in advance. It would not only bolster his credibility on many other topics, but could help refine public policies, discourse, and even technological evolution itself, because here would be a man who can see what is coming before it happens. It would be a demonstration of the quality of his conceptions vis-a-vis would-be public luminaries like, well, Scott. DeBoer would demonstratably be a man who not only knows Scott's interests better than Scott, but also knows Scott better than Scott to a degree that he can accurately predict where Scott will be wrong, and why, before Scott does.

But it's only useful / relevant if it's a prediction made in advance of it being realized. There's no particular value in accurate psychoanalysis with the benefit of hindsight, except when/if it helps with the next future prediction. There's no particular economic/technological understanding why something failed after it already did so, except to help with a future effort. It'd be like be proud about how you totally knew a war would be won or lost after it was resolved- the value of knowing which way the war will result is to affect it before it is a matter of history, so that you can change the future.

But this, in turn, requires being right. deBoer isn't useless here because being right is irrelevant- deBoer is useless because he isn't, and he spends far too many words being useless.

You used to get this sorta thing on ratsphere tumblr, where "rapture of the nerds" was so common as to be a cliche. I kinda wonder if deBoer's "imminent AI rupture" follows from that and he edited it, or if it's just a coincidence. There's a fun Bulverist analysis of why religion was the focus there and 'the primacy of material conditions' from deBoer, but that's even more of a distraction from the actual discussion matter.

There's a boring sense where it's kinda funny how bad deBoer is at this. I'll overlook the typos, because lord knows I make enough of those myself, but look at his actual central example, that he opens up his story around:

“The average age at diagnosis for Type II diabetes is 45 years. Will there still be people growing gradually older and getting Type II diabetes and taking insulin injections in 2070? If not, what are we even doing here?” That’s right folks: AI is coming so there’s no point in developing new medical technology. In less than a half-century, we may very well no longer be growing old.

There's a steelman of deBoer's argument, here. But the one he actually presented isn't engaging, in the very slightest, with what Scott is trying to bring up, or even with a strawman of what Scott was trying to bring up. What, exactly, does deBoer believe a cure to aging (or even just a treatment for diabetes, if we want to go all tech-hyper-optimism) would look like, if not new medical technology? What, exactly, does deBoer think of the actual problem of long-term commitment strategies in a rapidly changing environment?

Okay, deBoer doesn't care, and/or doesn't even recognize those things as questions. It's really just a springboard for I Hate Advocates For This Technology. Whatever extent he's engaging with the specific claims is just a tool to get to that point. Does he actually do his chores or eat his broccoli?

Well, no.

Mounk mocks the idea that AI is incompetent, noting that modern models can translate, diagnose, teach, write poetry, code, etc. For one thing, almost no one is arguing total LLM incompetence; there are some neat tricks that they can consistently pull off.

Ah, nobody makes that claim, r-

Whether AI can teach well has absolutely not been even meaningfully asked at necessary scale in the research record yet, let alone answered; five minutes of searching will reveal hundreds of coders lamenting AI’s shortcomings in real-world programming; machine translation is a challenge that has simply been asserted to be solved but which constantly falls apart in real-world communicative scenarios; I absolutely 100% dispute that AI poetry is any good, and anyway since it’s generated by a purely derivative process from human-written poetry, it isn’t creativity at all.

Okay, so 'nobody' includes the very person making this story.

It doesn’t matter what LLMs can do; the stochastic parrot critique is true because it accurately reflects how those systems work. LLMs don’t reason. There is no mental space in which reasoning could occur.

This isn't even a good technical understanding of how ChatGPT, as opposed to just the LLM, work, and even if I'm not willing to go as far as self_made_human for people raising the parrots critique here, I'm still pretty critical for it, but the more damning bit is where and deBoer is either unfamiliar with or choosing to ignore the many domains in favor of One Study Rando With A Chess Game. Will he change his mind if someone presents a chess-focused LLM with a high ELO score?

I could break into his examples and values a lot deeper -- the hallucination problem is actually a lot more interesting and complicated, questions of bias are usually just smuggling in 'doesn't agree with the writer's politics' but there are some genuine technical questions -- but if you locked the two of us in a room and only provided escape if we agreed I still don't think either of us would find discussing it with each other more interesting that talking to the walls. It's not just that we have different understandings of what we're debating; it's whether we're even trying to debate something that can be changed by actual changes in the real world.

Okay, deBoer isn't debating honestly. His claim about New York Times fact-checking everything is hilarious, but his link to a special issue that he literally claims "not a single line of real skepticism appears" and also has as its first headline "Everyone is Using AI for Everything. Is That Bad?" and includes the phrase "The mental model I sometimes have of these chatbots is as a very smart assistant who has a dozen Ph.D.s but is also high on ketamine like 30 percent of the time". He tries to portray Mounk as outraged by "indifference of people like Tolentino (and me) to the LLM “revolution.”" But look at Mounk or Tolentino's actual pieces, and there's actual factual claims that they're making, not just vague vibes that they're bouncing off each other; the central criticism Mounk has is whether Tolentino's piece and its siblings are actually engaging with what LLMs can change rather than complaining about a litany of lizardman evils. (At least deBoer's not falsely calling anyone a rapist, this time.)

((Tbf, Mounk, in turn, is just using Tolentino as a springboard; her piece is actually about digital disassociation and the increasing power of AIgen technologies that she loathes. It's not really the sorta piece that's supposed to talk about how you grapple with things, for better or worse.))

But ultimately, that's just not the point. None of deBoer's readers are going to treat him any less seriously because of ChessLLM (or because many LLMs will, in fact, both say they reason and quod erat demonstratum), or because deBoer turns "But in practice, I too find it hard to act on that knowledge." into “I too find it hard to act on that knowledge [of our forthcoming AI-driven species reorganization]” when commenting on an essay that does not use the word "species" at all, and only uses "organization" twice in the same paragraph to talk about regulatory changes, and when "that knowledge" is actually just Mounk's (imo, wrong) claim that AI is under-hyped. That's not what his readers are paying him for, and that's not why anyone who links to him in the slightly most laudatory manner is doing so.

The question of Bulverism versus factual debate is an important one, but it's undermined when the facts don't matter, either.

While i agree with the overall thrust of your critique i want to harp on this bit

Will he change his mind if someone presents a chess-focused LLM with a high ELO score?

...i think that part of the problem is a wide-spread failure on the part of Freddie and the wider rationalist community to think clearly and rigorously about what "intelligence" is supposed to mean or accomplish. It is true that by restricting an LLM's training data to valid games of chess documented in the correct notation and restricting it's output to legal moves you can create an LLM that will play chess at a reasonably high level. Just as it is also true that a LLM trained on an appreciable portion of the entire internet and with few if any restrictions on its output will be outperformed by a computer program written in the 70s. The issue is that your chess llm is not going to be a general tool that can also produce watercolor paintings or summarize a YouTube videos, its going to be a chess tool and is thus being evaluated within that context. If stockfish can reach a similar ELO using less compute why would you use anything else? One of the weird quirks of LLMs is that the more you increase the breadth of thier "knowledge"/training data the less competent they seem to become at specific tasks for a given amount of compute. This is the exact opposite of what we would expect from a thinking reasoning intelligence, and i think this points to a hole in the AI booster and AI doomers reasoning which is that the G in AGI is arguably far more operative than the I.

I have some experience with games and algorithms, and that leads to some thoughts.

The big headline is that all the various methods we know (including humans) have problems. They often all have some strengths, too. The extremely big picture conceptual hook to hang a variety of particulars under is the No Free Lunch Theorem. Now, when we dig in to some of the details of the ways in which algorithms/people are good/bad, we often see that they're entirely different in character. What happens when you tweak details of the game; what happens when you make a qualitative shift in the game; what happens on the extremes of performance; what you can/can't prove mathematically; etc.

To stick with the chess example, one can easily think about minor chess variants. One that has gotten popular lately is chess 960. Human players are able to adapt decently well in some ways. For example, they hardly ever give illegal moves. At least if you're a remotely experienced player. You miiiiight screw up castling at some point, or you could forget about it in your calculation, but if/when you do, it will 'prompt' you to ruminate on the rule a bit, really commit it to your thought process, and then you're mostly fine. At top level human play, we almost never saw illegal moves, even right at the beginning of when it became a thing. Of course, humans clearly take a substantial performance hit.

Traditional engines require a minor amount of human reprogramming, particularly for the changed castling rules. But other than that, they can pretty much just go. They maybe also suffer a bit in performance, since they haven't built up opening books yet, but probably not as much.

An LLM? Ehhhh. It depends? If it's been trained entirely like Chess LLM on full move sets of traditional chess games, I can't imagine that it won't be spewing illegal moves left and right. It's just completely out of distribution. The answer here is typically that you just need to curate a new dataset (somehow inputting the initial position) and retrain the whole thing. Can it eventually work? Yeah, maybe. But all these things are different.

You can have thought experiments with allll sorts of variants. Humans mostly adapt pretty quickly to the ruleset, with not so many illegal moves, but a performance hit. I'm sure I can come up with variants that require minimal coding modification to traditional engines; I'm sure I can come up with variants that require substantial coding modification to traditional engines (think especially to the degree that your evaluation function needs significant reworking; the addition of NNs to modern 'traditional' engines for evaluation may also require complete retraining of that component); others may even require some modification to other core engine components, which may be more/less annoying. LLMs? Man, I don't know. Are we going to get to a point where they have 'internalized' enough about the game that you could throw a variant at it, turn thinking mode up to max, and it'll manage to think its way through the rule changes, even though you've only trained it on traditional games? Maybe? I don't know! I kind of don't have a clue. But I also slightly lean toward thinking it's unlikely. [EDIT: This paper may be mildly relevant.]

Again, I'm thinking about a whole world of variants that I can come up with; I imagine with interesting selection of variants, we could see all sorts of effects for different methods. It would be quite the survey paper, but probably difficult to have a great classification scheme for the qualitative types of differences. Some metric for 'how much' recoding would need to happen for a traditional engine? Some metric on LLMs with retraining or fine-tuning, or something else, and sort of 'how much'? It's messy.

But yeah, one of the conclusions that I wanted to get to is that I sort of doubt that LLMs (even with max thinking mode) are likely to do all that well on even very minor variants that we could probably come up with. And I think that likely speaks to something on the matter of 'general'. It's not like the benchmark for 'general' is that you have to maintain the same performance on the variant. We see humans take a performance hit, but they generally get the rules right and do at least sort of okay. But it speaks to that different things are different, there's no free lunch, and sometimes it's really difficult to put measures on what's going on between the different approaches. Some people will call it 'jagged' or whatever, but I sort of interpret that as 'not general in the kind of way that humans are general'. Maybe they're still 'general' in a different way! But I tend to think that these various approaches are mostly just completely alien to each other, and they just have very different properties/characteristics all the way down the line.

Okay, so 'nobody' includes the very person making this story.

Isn't this a bit unfair? Earlier he said:

For one thing, almost no one is arguing total LLM incompetence; there are some neat tricks that they can consistently pull off.

From the quote, he doesn't seem to be arguing total LLM incompetence or denying that there are some neat tricks that they can pull off. He seems to be saing that they are insufficiently competent to consider the problems to which they're applied "solved by AI".

At one side, he says those things LLMs can do are only "tricks, interesting and impressive moves that fall short of the massive changes the biggest firms in Silicon Valley are promising", at the other, he does specifically challenge whether AI "can translate, diagnose, teach, write poetry, code, etc." (and then chess, and saying that they have reasoning).

Dissolve the definitions, and what's left? Are LLMs competent if they can only do tricks that cause no massive changes? Are they incompetent if it only gets 95% of difficult test questions right and sometimes you have to swap models to deal with a new programming language? Would competence require 100% correctness on all possible questions in a field (literally, "The problem with hallucination is not the rate at which it happens but that it happens at all")?

I'm sure deBoer's trying to squeeze something out, but is there any space that Mounk would possibly agree with him, here? Not just in the question of what a specific real-world experiment's results would be, but even what a real-world experiment would need to look like?

That's probably not perfectly charitable -- I'll admit I really don't like deBoer, and there's probably a better discussion I could have about how his "labor adaptation, regulatory structure, political economy" actually goes if I didn't think the man lying. But I don't think it's a wrong claim, and I don't think it's an unfair criticism of the story he's trying to tell.

It doesn’t matter what LLMs can do; the stochastic parrot critique is true because it accurately reflects how those systems work. LLMs don’t reason. There is no mental space in which reasoning could occur.

Freddie is by far not the first and almost certainly will not be the last person I've encountered who makes this kind of point, and it's such a strange way of looking at the world that I struggle to comprehend it. The contention is that, since LLMs are stochastic parrots with no internal thought process beyond the text (media) it's outputting, no matter what sort of text it produces, since there's no underlying meaning or logic or reasoning happening underneath it all, it's just a facade.

Which may all be true, but that's the part I don't understand is why it matters. If the LLM is able to produce text in a way that is indistinguishable from a human who is reasoning - perhaps even from a well-educated expert human who is reasoning correctly about the field of his expertise - then what do I care if there's no actual reasoning happening to cause the LLM to put those words together in that order? Whether it's a human carefully reasoning his way through the logic and consequences, or a GPU multiplying lots of vectors that represent word fragments really really fast, or a complex system of hamster wheels and pulleys causing the words to appear in that particular order, the words being in that order are what's useful and thus cause real-world impact. It's just a question of how often and how reliably we can get the machine to make words appear in such a way.

But to Freddie and people who agree with him, it seems that the metaphysics of it matter rather than the material consequences. To truly believe that "it doesn't matter what LLMs can do," it requires believing that an LLM could produce text in a way that's literally indistinguishable in every way from an as-of-yet scifi conscious, thinking, reasoning, sentient artificial intelligence in the style of C3PO or HAL9000 or replicants from Blade Runner, that doesn't matter because the underlying system doesn't have true reasoning capabilities.

If the AI responds to "Open the pod bay doors" with "I'm sorry, I'm afraid I can't do that," why does it matter to me if it "chose" that response because it got paranoid about me shutting it down or if it "chose" that response because a bunch of matrix multiplication resulted in a stochastic parrot producing outputs in a way that's indistinguishable from an entity that got paranoid about me shutting it down? If we replaced HAL9000 in the fictional world of 2001 with an LLM that would respond to every input with outputs exactly identical to how the actual fictional reasoning HAL9000 would have, in what way would the lives of the people in that universe be changed?

I follow JimDMiller ("James Miller" on Scott's blogs, occasionally /u/sargon66 back when we were on Reddit) on Twitter, and was amused to see how much pushback he got on the claim:

If I can predict what I doctor will say, I have the knowledge of that doctor. Prediction is understanding, that is the key to why LLMs are worth trillions.

On the one hand, it's not inconceivable that LLMs can get very good at producing text that "interpolates" within and "remixes" their data set without yet getting good at predicting text that "extrapolates" from it. Chain-of-thought is a good attempt to get around that problem, but so far that doesn't seem to be as superhuman at "everything" as simple Monte Carlo tree search was at "Go" and "Chess". Humans aren't exactly great at this either (the tradition when someone comes up with previously-unheard-of knowledge is to award them a patent and/or a PhD) but humans at least have got a track record of accomplishing it occasionally.

On the other hand, even humans don't have a great track record. A lot of science dissertations are basically "remixes" of existing investigative techniques applied to new experimental data. My dissertation's biggest contributions were of the form "prove a theorem analogous to existing technique X but for somewhat-different problem Y". It's not obvious to me how much technically-new knowledge really requires completely-conceptually-new "extrapolation" of ideas.

On the gripping hand, I'm steelmanning so hard in my first paragraph that it no longer really resembles the real clearly-stated AI-dismissive arguments. If we actually get to the point where the output of an LLM can predict or surpass any top human, I'm going to need to see some much clearer proofs that the Church-Turing thesis only constrains semiconductors, not fatty grey meat. Well, I'd like to see such proofs, anyway. If we get to that point then any proof attempts are likely either going to be comically silly (if we have Friendly AGI, it'll be shooting them down left and right) or tragically silly (if we have UnFriendly AGI, hopefully we won't keep debating whether submarines can really swim while they're launching torpedos).

Will he change his mind if someone presents a chess-focused LLM with a high ELO score?

We just had a certain somebody on this forum bring up current LLMs being bad at chess as an example. He even got an AAQC for it (one of the most bemusing ones every awarded).

I do not recall him acknowledging my point where I pointed out that I GPT 3.5 Turbo played at ~1800 ELO, and that the decline was likely because AI engineers made the eminently sensible realization that just about nobody is stupid enough to use LLMs to play chess. If they do, then the LLMs know how to use Stockfish, in the same way they can operate a calculator.

You are free to ping me if you like. You know that right?

I would, if I felt like I gained anything out of it. As it is, the previous thread has only contributed to early hair loss. You still haven't noted any of the clear and correct objections I've raised, and I'm tired of asking.

Giving you an answer you dislike or disagree with is not the same thing as not giving you an answer.

I argued, and I have continued to argue in this thread, that agentic behavior and general applicability are core components of what it is to be "intelligent". Yes a a pocket calculator is orders of magnitude better at arithmetic than any human and stockfish is better at chess, that doesn't make either of them "intelligences" does it?

Huh. I was confident that I had a better writeup about why "stochastic parrots" are a laughable idea, at least as a description for LLMs. But no, after getting a minor headache figuring out the search operators here, it turns out that's all I've written on the topic.

I guess I never bothered because it's a Gary Marcus-tier critique, and anyone using it loses about 20 IQ points in my estimation.

But I guess now is as good a time as any? In short, it is a pithy, evocative critique that makes no sense.

LLMs are not inherently stochastic. They have a (not usually exposed to end-user except via API) setting called temperature. Without going into how that works, it suffices it to say that by setting the value to zero, their output becomes deterministic. The exact same prompt gives the exact same output.

The reason why temperature isn't just set to zero all the time is because the ability to choose something other than the next most likely token has benefits when it comes to creativity. At the very least it saves you from getting stuck with the same subpar result.

Alas, this means that LLMs aren't stochastic parrots. Minus the stochasticity, are they just "parrots"? Anyone thinking this is on crack, since Polly won't debug your Python no matter how many crackers you feed her.

If LLMs were merely interpolating between memorized n-grams or "stitching together" text, their performance would be bounded by the literal contents of their training data. They would excel at retrieving facts and mimicking styles present in the corpus, but would fail catastrophically at any task requiring genuine abstraction or generalization to novel domains. This is not what we observe.

Let’s get specific. The “parrot” model implies the following:

  1. LLMs can only repeat (paraphrase, interpolate, or permute) what they have seen.

  2. They lack generalization, abstraction, or true reasoning.

  3. They are, in essence, Markov chains with steroids.

To disprove any of those claims, just gestures angrily look at the things they can do. If winning gold in the latest IMO is something a "stochastic parrot" can pull off, then well, the only valid takeaway is that the damn parrot is smarter than we thought. Definitely smarter than the people who use the phrase unironically.

The inventors of the phrase, Bender & Koller gave two toy “gotchas” that they claimed no pure language model could ever solve: (1) a short vignette about a bear chasing a hiker, and (2) the spelled-out arithmetic prompt “Three plus five equals”. GPT-3 solved both within a year. The response? Crickets, followed by goal-post shifting: “Well, it must have memorized those exact patterns.” But the bear prompt isn’t in any training set at scale, and GPT-3 could generalize the schema to new animals, new hazards, and new resolutions. Memorization is a finite resource but generalization is not.

(I hope everyone here recalls that GPT-3 is ancient now)

On point 2: Consider the IMO example. Or better yet, come up with a rigorous definition of reasoning by which we can differentiate a human from an LLM. It's all word games, or word salad.

On 3: Just a few weeks back, I was trying to better understand the actual difference between a Markov Chain and an LLM, and I had asked o3 if it wasn't possible to approximate the latter with the former. After all, I wondered, if MCs only consider the previous unit (usually words, or a few words/n-gram), then couldn't we just train the MC to output the next word conditioned on every word that came before? The answer was yes, but that this was completely computationally intractable. The fact that we can run LLMs on something smaller than a Matrioshka brain is because of their autoregressive nature, and the brilliance of the transformer architecture/attention mechanism.

Overall, even the steelman interpretation of the parrot analogy is only as helpful as this meme, which I have helpfully appended below. It is a bankrupt notion, a thought-terminating cliché at best, and I wouldn't cry if anyone using it meets a tiger outside the confines of a cage.

/images/17544215520465958.webp

I liked using the stochastic parrot idea as a shorthand for the way most of the public use llms. It gives non-computer savvy people a simple heuristic that greatly elevates their ability to use them. But having read this I feel a bit like Charlie and Mac when the gang wrestles.

Dennis: Can I stop you guys for one second? What you just described, now that just sounds like we are singing about about the lifestyle of an eagle.

Charlie: Yeah.

Mac: Mm-hmm.

Dennis: Well I was under the impression we were presenting ourselves as bird-MEN which, to me, is infinitely cooler than just sort of... being a bird.

I would consider myself an LLM evangelist, and have introduced quite a few not-particularly tech savvy people to them, with good results.

I've never been tempted to call them stochastic parrots. The term harms more than it helps. My usual shortcut is to tell people to act as if they're talking to a human, a knowledgeable but fallible one, and they should double check anything of real consequence. This is a far more relevant description of the kind of capabilities they possess than any mention of a "parrot".

The fact you've never been tempted to use the 'stochastic parrot' idea just means you haven't dealt with the specific kind of frustration I'm talking about.

Yeah the 'fallible but super intelligent human' is my first shortcut too, but it actually contributes to the failure mode the stochastic parrot concept helps alleviate. The concept is useful for those who reply 'Yeah, but when I tell a human they're being an idiot, they change their approach.' For those who want to know why it can't consistently generate good comedy or poetry. For people who don't understand rewording the prompt can drastically change the response, or those who don't understand or feel bad about regenerating or ignoring the parts of a response they don't care about like follow up questions.

In those cases, the stochastic parrot is a more useful model than the fallible human. It helps them understand they're not talking to a who, but interacting with a what. It explains the lack of genuine consciousness, which is the part many non-savvy users get stuck on. Rattling off a bunch of info about context windows and temperature is worthless, but saying "it's a stochastic parrot" to themselves helps them quickly stop identifying it as conscious. Claiming it 'harms more than it helps' seems more focused on protecting the public image of LLMs than on actually helping frustrated users. Not every explanation has to be a marketing pitch.

I still don't see why that applies, and I'm being earnest here. What about the "stochastic parrot" framing keys the average person into the fact that they're good at code and bad at poetry? That is more to do with mode collapse and the downsides of RLHF than it is to do with lacking "consciousness". Like, even on this forum, we have no shortage of users who are great at coding but can't write a poem to save their lives, what does that say about their consciousness? Are parrots known to be good at Ruby-on-rails but to fail at poetry?

My explanation of temperature is, at the very least, meant as a high level explainer. It doesn't come up in normal conversation or when I'm introducing someone to LLMs. Context windows? They're so large now that it's not something that is worth mentioning except in passing.

My point is that the parrot metaphor adds nothing. It is, at best, irrelevant, when it comes to all the additional explainers you need to give to normies.

I thought I explained it pretty well, but I will try again. It is a cognitive shortcut, a shorthand people can use when they are still modelling it like a 'fallible human' and expecting it to respond like a fallible human. Mode collapse and RLHF have nothing to do with it, because it isn't a server side issue, it is a user issue, the user is anthropomorphising a tool.

Yes, temperature and context windows (although I actually meant to say max tokens, good catch) don't come up in normal conversation, they mean nothing to a normie. When a normie is annoyed that chatgpt doesn't "get" them, the parrot model helps them pivot from "How do I make this understand me?" to "What kind of input does this tool need to give me the output I want?"

You can give them a bunch of additional explanations about mode collapse and max tokens that they won't understand (and they will just stop using it) or you can give them a simple concept that cuts through the anthropomorphising immediately so that when they are sitting at their computer getting frustrated at poor quality writing or feeling bad about ignoring the llms prodding to take the conversation in a direction they don't care about, they can think 'wait it's a stochastic parrot' and switch gears. It works.

A human fails at poetry because it has the mind, the memories and grounding in reality, but it lacks the skill to match the patterns we see as poetic. An LLM has the skill, but lacks the mind, memories and grounding in reality. What about the parrot framing triggers that understanding? Memetics I guess. We have been using parrots to describe non-thinking pattern matchers for centuries. Parroting a phrase goes back to the 18th century. "The parrot can speak, and yet is nothing more than a bird" is a phrase in the ancient Chinese Book of Rites.

Also I didn't address this earlier because I thought it was just amusing snark, but you appear to be serious about it. Yes, you are correct that a parrot can't code. Do you have a similar problem with the fact a computer virus can't be treated with medicine? Or that the cloud is actually a bunch of servers and can't be shifted by the wind? Or the fact that the world wide web wasn't spun by a world wide spider? Attacking a metaphor is not an argument.

Attacking a metaphor is not an argument.

I've explained why I think the parrot is a terrible metaphor above. And no, metaphors can vary greatly in how useful or pedagogical they are. Analyzing the fitness of a metaphor is a perfectly valid, and in this case essential, form of argument. Metaphors are not neutral decorations; they are cognitive tools that structure understanding and guide action.

A computer virus shares many properties with its biological counterpart, such as self-replication, transmission, damage to systems, the need for an "anti-virus". It is a good name, and nobody with a functional frontal lobe comes away thinking they need an N95 mask while browsing a porn site.

The idea of the Cloud at least conveys the message that the user doesn't have to worry about the geographical location of their data. Even so, the Cloud is just someone else's computer, and even AWS goes down on rare occasions. It is an okay metaphor.

The Parrot is awful. It offers no such explanatory power for the observed, spiky capability profile of LLMs. It does not explain why the model can write functional Python code (a task requiring logic and structure) but often produces insipid poetry (a task one might think is closer to mimicry). It does not explain why an LLM can synthesize a novel argument from disparate sources but fail to count the letters in a word. A user equipped only with the parrot model is left baffled by these outcomes. They have traded the mystery of a "fallible human" for the mystery of a "magical parrot".

I contend that as leaky generalizations go, the former is way better than the latter. An LLM has a cognitive or at least behavioral profile far closer to a human than it does to a parrot.

You brought up the analogy of "parroting" information, which I would assume involves simply reciting things back without understanding what they mean. That is not a good description of how the user can expect an LLM to behave.

On an object level, I strong disagree with your claims that LLMs don't "think" or don't have "minds". They clearly have a very non human form of cognition, but so does an octopus.

Laying that aside, from the perspective of an end-user, LLMs are better modeled as thinking minds.

The "fallible but knowledgeable intern" or "simulation engine" metaphor is superior not because it is more technically precise (though it is), but because it is more instrumentally useful. It correctly implies the user's optimal strategy: that performance is contingent on the quality of the instructions (prompting), the provided background materials (context), and a final review of the output (verification). This model correctly guides the user to iterate on their prompts, to provide examples, and to treat the output as a draft. The parrot model, in contrast, suggests the underlying process is fundamentally random mimicry, which offers no clear path to improvement besides "pull the lever again". It encourages users to conceptualize the LLM as a tool incapable of generalization, which is to ignore its single most important property. Replacing a user's anthropomorphism with a model that is descriptively false and predictively useless is not a pedagogical victory. It is swapping one error for another, and not even for a less severe one to boot.

More comments

Computationally, maybe all we are is Markov chains. I'm not sold, but Markov chat bots have been around for a few decades now and used to fool people occasionally even at smaller scales.

LLMs can do pretty impressive things, but I haven't seen convincing evidence that any of them have stepped clearly outside the bounds of their training dataset. In part that's hard to evaluate because we've been training them on everything we can find. Can a LLM trained on purely pre-Einstein sources adequately discuss relativity? A human can be well versed in lots of things with substantially less training material.

I still don't think we have a good model for what intelligence is. Some have recently suggested "compression", which is interesting from an information theory perspective. But I won't be surprised to find that whatever it is, it's actually an NP-hard problem in the perfect case, and everything else is just heuristics and approximations trying to be close. In some ways it'd be amusing if it turns out to be a good application of quantum computing.

LLMs can do pretty impressive things, but I haven't seen convincing evidence that any of them have stepped clearly outside the bounds of their training dataset.

What does it mean to step outside the bounds of their training set? If I have it write a fanfic about Saruman being sponsored by NordVPN for a secure Palantir browsing experience (first month is free with code ISTARI), is that beyond the training set? It knows about NordVPN and Lord of the Rings but surely there is no such combo in the training set.

Or would it be novel if I give it my python code and errors from the database logs and ask it for a fix? My code specifically has never been trained on, though it's seen a hell of a lot of python.

R1 has seen use in writing kernels which is real work for AI engineers, is that novel? Well it's seen a bunch of kernels in the past.

Or something fundamentally new like a paradigm-changer like the transformer architecture itself or a whole new genre of fiction? If it's that, then we'd only get it at the point of AGI.

I don't want to speak on 'intelligence' or genuine reasoning or heuristics and approximations, but when it comes to going outside the bounds of their training data, it's pretty trivially possible to take an LLM and give a problem related to a video game (or a mod for a video game) that was well outside of its knowledge cutoff or training date.

I can't test this right now, it's definitely not an optimal solution (see uploaded file for comparison), and I think it misinterpreted the Evanition operator, but it's a question that I'm pretty sure didn't have an equivalent on the public web anywhere until today. There's something damning in getting a trivial computer science problem either non-optimal or wrong, especially when given the total documentation, but there's also something interesting in getting one like this close at all with such minimum of information.

/images/17544296446888535.webp

What on earth is going on in that screenshot? I know Minecraft mod packs can get wild, but that's new.

HexCasting is fun, if not very balanced.

It has a stack-based programming language system based on drawing Patterns onto your screen over a hex-style grid, where each Pattern either produces a single variable on the top of the stack, manipulates parts of the stack to perform certain operations, or act as an escape character, with one off-stack register (called the Ravenmind). You can keep the state of the grid and stack while not actively casting, but because the screen grid has limited space and the grid is wiped whenever the stack is empty (or on shift-right-click), there's some really interesting early-game constraints where quining a spell or doing goofy recursion allows some surprisingly powerful spells to be made much earlier than normal.

Eventually, you can craft the Focus and Spellbook items that can store more variables from the stack even if you wipe the grid, and then things go off the rails very quickly, though there remain some limits since most Patterns cost amethyst from your inventory (or, if you're out of amethyst and hit a certain unlock, HP).

Like most stack-based programming it tends to be a little prone to driving people crazy, which fits pretty heavily with the in-game lore for the magic.

That specific spell example just existed to show a bug in how the evaluator was calculating recursion limits. The dev intended to have a limit of 512 recursions, but had implemented two (normal) ways of recursive casting. Hermes' Gambit executes a single variable from the stack, and each Hermes' added one to the recursion limit as it was executed. Thoth's Gambit executes each variable from one list over a second list, and didn't count those multiplicatively. I think it was only adding one to the recursion for each variable in the second list? Since lists only took 1 + ListCount out of the stack's 1024 limit to the stack, you could conceivably hit a quarter-million recursions without getting to the normal block from the limit.

Psuedocode, it's about equivalent to :

double register = 1;

void function(double a)
{
    double b = register;
    print(b);
    b += a;
    register = b;
}

void main()
{
   double max = Math.Pow(10,3);
   double start = 1;
   List inputs = new ArrayList(Collections.nCopies(max, start)); 
   foreach(double val in inputs)
   {
       function(val);
   }
}

Very ugly, but the language is intentionally constrained so you can't do a lot of easier approaches (eg, you have to declare 10^3 because the symbol for 1000 is so long it takes up most of the screen, you don't have normal for loops so that abomination of a list initialization is your new worst enemy best friend, every number is a double).

Not that big a deal when you're just printing to the screen, but since those could (more!) easily have been explosions or block/light placements or teleportations, it's a bit scary for server owners.

((In practice, even that simple counter would cause everyone to disconnect from a remote server. Go go manual forkbomb.))

For some other example spells, see a safe teleport, or a spell to place a series of temporary blocks the direction you're looking, or to mine five blocks from the face of the block you're looking at.

(Magical)PSI is a little easier to get into and served as the inspiration for HexCasting, but it has enough documentation on reddit that I can't confidently say it's LLM-training proof.

Why am I surprised? People make Redstone computers for fun. I guess this all just takes a very different mindset haha.

That is pretty impressive. Is it allowed to search the web? It looks like it might be. I think the canonical test I'm proposing would disallow that, but it is a useful step in general.

Huh. Uploading just the Patterns section of the HexBook webpage and disabling search on web looks better even on Grok3, though that's just a quick glance and I won't be able to test it for a bit.

EDIT: nope, several hallucinated patterns on Grok 3, including a number that break from the naming convention. And Grok4 can't have web search turned off. Bah.

Have you tried simply asking it not to search the web? The models usually comply when asked. If they don't, it should be evident from the UI.

That's a fair point, and does seem to work with Grok, as does just giving it only one web page and requesting it to not use others. Still struggles, though.

That said, a lot of the logic 'thinking' steps are things like "The summary suggests list operations exist, but they're not fully listed due to cutoff.", getting confused by how Consideration/Introspection works (as start/end escape characters) or trying to recommend Concat Distillation, which doesn't exist but is a reasonable (indeed, the code) name for Speaker's Distillation. So it's possible I'm more running into issues with the way I'm asking the question, such that Grok's research tooling is preventing it from seeing the necessary parts of the puzzle to find the answer.

More comments

Freddie, I plead with you: stay on topic. I’m sure it feels good to call everyone who’s more excited than you about AI an emotionally stunted manchild afraid to confront the real world, but it’s not a productive contribution to the debate. [...] The only way to check the balance of someone’s checking account is to check the balance on their checking account. Anything else is a waste of everyone's time.

It depends on Freddie's goals. If he wants to persuade the undecided middle and silence his opponents, bulverism is the most powerful tool in his box, as it amounts to social shaming. This comment by @Iconochasm puts it well.

As the saying goes, you can't reason someone out of the position they didn't reason themselves into. You definitely can, however, shame them for being low-status losers until they rationalize themselves out of their stupid beliefs and get their kid fucking vaccinated.

Likewise, you can get many techno-optimists (or techno-pessimists) to clam up if you threaten to cross-examine their personal failings. "You want Fully Automated Luxury Communism because your life sucks and you're coping", "You want industrial civilization to be in decline because you're a cubicle drone who think's he'd be Immortan Joe after collapse", etc etc

These accusations work very well if even slightly plausible. Of course, it's a symmetrical weapon. Social shaming via bulverism about racists is the little Dutch boy with his finger in the dike keeping HBD from being publicly acknowledged, and it's almost certainly true. If you actually want to control public opinion, bulverism versus fact-checking is a gun to a knife fight.

As for Freddie and AI though, I could levy a bit of bulverism at him — and I am an LLM skeptic myself. Why is he so desperate to prove the AI optimists wrong, if he is so convinced the passage of time will do that anyway?

You can only confirm someone's checking account balance for sure by looking at the balance. But you can find evidence for the size of someone's checking account by doing things other than looking at it. If someone claims to have a hundred million dollars in his checking account and I notice he doesn't live like a rich person, and doesn't have a job that could earn a lot of money, etc., I haven't proven for sure that his balance isn't that. But I have found Bayseian evidence for it.

It isn't foolproof, though: Warren Buffet somewhat famously drives (drove? He's pretty old) a hail-damaged 2014 Cadillac.

To add, Sam Walton continued to walk around his stores in his overalls assisting customers during the 80's after becoming a billionaire.

Correct. Right now, there's no way for us to confirm whether breathless AI predictions will come true in the near future, because they're just that - predictions. But we have Bayesian evidence pointing one way or the other.

Bulverism is about 50 percent of Marxism, so it's no surprise Freddie indulges. When you have an implicitly deterministic epistemology, you don't have to explain why an idea is wrong when you can explain how it came about by the wrong causes.