This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

Jump in the discussion.
No email address required.
Notes -
The singularity is coming in 2027 and no one cares.
Dwarkesh is out with a new essay in which he argues that there is a high chance (not 100%, but like 75%) that we get an intelligence explosion in the next 12 months equal in magnitude to the jump between GPT-3.5 and GPT-6.
This jump would be caused by the automation of AI research.
I highly, highly recommend giving this a read or a listen. Dwarkesh is, in my mind, the most thoughtful public intellectual around today (sorry Scott) and is known for his in depth research and thoughtful interviews. In his essay, it's likely that he addresses whatever objections that you have that just sprung to mind, so just give it a read. Really.
At this point, it seems the only thing that would stop superintelligence from forming is an unknown unknown, as in there is some fundamental constraint or that hasn't been identified, or an emergent phenomenon that only reveals itself in the future. I guess this has all been rehashed a million times, and I don't have a lot to add beyond what Dwarkesh said, so I'll revert to the purpose of this board which is to make some social commentary.
People are completely unprepared for what's coming.
In financial forums that I visit, and in youtube videos I watch, people are entirely dismissive of what is happening at OpenAI and Anthropic. "It doesn't really think! They're losing money! Drug development, no way! It only works for software! The software is buggy! They'll be bankrupt in a year! Switching costs are zero! Cheap Chinese models will win!".
Etc.. Etc... Etc...
Some concepts are just too high IQ for most people. Probably most people with an IQ below 100 don't really understand supply and demand for example. And the floor for understanding recursive self improvement is a lot higher.
Everything feels pretty normal now. But there's at some point going to be a big oh shit moment when social consensus hits and 90% of the population goes from denialism to doomerism in a short period. I have a hard time predicting what will happen then except that I'd assign a 50% chance that Anthropic will be the most valuable company on Earth in 12 months. Cold comfort. Superintelligence is coming and it's going to change everything. Maybe it will be good...
This isn't my read on things. Average and dumb people can understand the concept just fine. They might not like it and the changes it brings, but they absolutely recognize the threat; it's been communicated by, if nothing else, pretty simple and direct sci fi stories that they've seen on the big screen.
But it takes a higher IQ person to come up with superficially insightful but wrong/irrelevant points that leads to true AI denialism (as opposed to AI opposition). Stochastic parrots, Chinese rooms, explanations of why a trend has to stop. They've heard of P =! NP; surely that prevents AI! They can construct a TESCREAL genealogy of eugenics to AI, therefore it's bad and fake. Etc.
There are stronger arguments, of course, but I'd classify questions around things like model performance generalizing beyond RLVR friendly tasks as a different category than true denialism.
Right - I think the OP comes off a bit too much like "most people aren't as smart as me", whereas I suspect that most people understand the central plot point of The Terminator.
"We build a superintelligent computer and things go badly" isn't some sort of weird esoterica. It's been a theme in popular media for over half a century. You don't need a high IQ to understand, "Maybe we build a computer and the computer improves itself and then the computer destroys us".
Charitably, maybe the OP means that most people don't think that's likely in reality, or that most people don't understand the specific mechanism that he posits for it. I think the former is probably true and the latter is probably false. I think most people don't expect Skynet to appear, but most people probably can understand the concept of a computer that makes itself smarter. It's really not a very complicated idea.
It's weird that it doesn't really figure into popular ideas of AI, though, isn't it?
In elite non-fiction we get "Speculations Concerning the First Ultraintelligent Machine" in 1965.
In popular fiction, just looking at your linked comment: you're right that "AI might be dangerous" is well-attested in popular fiction, but "AI might upgrade its intellectual capabilities extremely rapidly" not so much. WOPR/Joshua and HAL are dangerous mostly because of how smart they aren't, and Skynet and AM gain self-awareness but don't obviously get much smarter. There's a TNG episode where Barclay both gets much smarter and integrates with the computer, but it's all external influence, no runaway positive-feedback loop involved. Data is a little more realistic than C3-PO (the droid who's basically sapient but who isn't even obsolete decades later, much less creating his own successors), but it's still not surprising when Lal doesn't work and his research stops there.
The earliest popular fiction I know of featuring recursive self-improvement is A Fire Upon the Deep, so of course I went to the TVTropes page to see what the trope name was and if the trope page listed anything earlier ... and the only trope I see for the Blight is "Eldritch Abomination", nothing more specific. Even in that case, the self-improvement is essentially a "reboot" of previously-existing capabilities; in a sense the main interaction of the story with the idea of recursive self-improvement is that the "zones of thought" are a necessary gimmick to create a plausible future setting where superintelligence is a big deal but RSI isn't.
"AI capability is a function of accumulated research, and if that function's growth rate isn't slow enough then after AI can do any AI research y'=f(y) blows up" ... simple enough to squeeze into two clauses (for anyone who's had Calc 1; for "most people" maybe add a second or third sentence too), sure. But if you then add "we don't have reason to believe its growth rate will be slow before it hits far-superhuman levels" and "artificial-intelligence research is probably easier to do right than artificial-ethics research" then this has obvious implications of the Skynet variety, yet it's surprisingly hard to find people who were taking this seriously who didn't get there via Bostrom and Yudkowsky.
I think that's already overcomplicating it. The mathematics isn't important. What you're trying to get across is the idea of a learning supercomputer, and I think that's easily comprehensible by most people on the street.
Don't be distracted by the acronym 'RSI'. All 'RSI' means is "a computer that gets smarter very very quickly". I think that concept is both very simple and very well-understood.
I actually don't rate Bostrom or Yudkowsky as serious thinkers to begin with.
Who would you rate as a serious thinker then? They seem to me to have defined the age.
I wouldn't agree with that, but before explaining why, we should distinguish serious thinkers from impactful thinkers. 'Defining the age', or having a large effect on others or on the world, do not make a thinker serious. A serious thinker might be someone no one has ever heard of; a very influential thinker might be a superficial fool.
Now in this case I don't think Yudkowsky actually qualifies as either, but to be clear, I meant seriousness, not impact. The case is fairly self-explanatory in terms of impact - for all that Yudkowsky was and is a self-appointed 'expert' on AI, the industry as it actually came to exist has not been shaped by his ideas or prescriptions. The AI safety movement itself largely fails to meet the bar of effectiveness or impact, but even if it did, it does not function in a particularly Yudkowskyan way. I'm not sure what you mean by 'the age', but I do not see any significant way in which his ideas have shaped the world. Yudkowsky is a mildly famous blogger. That's all.
Still, my criticism was of the merits of Eliezer Yudkowsky Thought (and Nick Bostrom Thought), rather than of them as activists. Let me put it this way. What are their philosophically interesting ideas? What insights would you attribute to them? I'm not sure where to start, really - the Sequences are large, rambling, and turgid, and I would say not philosophically interesting, in the sense that very little in them is new or insightful.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Technically, an AI learning emotional intelligence and growing indistinguishable from a human would be a kind of self-improvement.
Though the kind of thing you are talking about is indeed rare. It would make for a pretty good villain if you had a AI that started out as dumber than the average human but over the course of the story increases its capabilities to become almost godlike by the end.
More options
Context Copy link
Her involves the AIs explicitly engaging in self improvement until they ultimately decide to go rogue and abandon humanity (a relatively optimistic outcome).
More options
Context Copy link
There are some things that come close. Like Arnie's explanation of Skynet's history in Terminator 2, "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 AM Eastern Time, August 29th."
There is also the TNG episode where in response to being asked to create a holodeck program capable of posing a challenge to Data's intelligence, the Enterprise's computer creates a holodeck program, Professor Moriarty, that is so smart that the crew only barely manage to defeat it. It's not explained whether the ship's computer was always that smart, or whether it did something novel in response to the prompt. Still suggestive, though.
I don't think it's really explored in the series, but the difference between Data, a self-aware android and character, and the ship's computer, which seems on par with Claude (or so) feels particularly salient. Maybe at the time it was a convenient tool for the writers, but I'm not sure how I'd describe it even. For lack of a better term, Data is shown to have a soul, but the computer does not. All sides seem to have lots of strong opinions on which our current models more strongly resemble.
I wouldn't say that Data is on par with Claude. Data is clearly a person. He has memories, desires, persistent self-awareness, and so on. Though Data claims to not have emotions, like Spock before him, he clearly does. Data is a human-like intelligence. Claude is not an intelligence at all.
From the writing perspective, I think it's fairly straightforward? Data is first and foremost a character. He is written as a person who happens to be made of metal, much as Worf is a person who happens to be a klingon. The ship's computer is not a character. It is a tool and presented as such. Everything about what Data is or how he functions is downstream of Data as a character,
You misread: he didn’t say Data was on par with Claude, he said the ship’s computer was.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Yeah, I've met people who insist that things like the Halting Problem and Godel's Incompleteness Theorem preclude the possibility of AI.
I didn't know Roger Penrose had schizophrenia.
Unfair, because he isn't in fact alone on this point, but it is pretty close. I'm pretty sure a frontier model can understand the standard proof of Godel's theorem already.
I think what you are trying to say here is that Roger Penrose is pretty much the only person making this argument about the Halting Problem and Godel's Incompleteness Theorem. However, I would appreciate it if you would make your point more plain.
In any event, if a prominent person makes an argument, it's very likely that a lot of non-prominent people will make the same argument. Especially if that argument is seductive or otherwise appealing to peoples' vanity.
I'm not sure how you would test this, but for me, the key point is that if there is some theoretical limit on the capabilities of AI, the natural assumption should be that the same fundamental limit applies to human thinking. Those who claim otherwise bear the burden of proof and so far I haven't seen any actual proof.
More options
Context Copy link
More options
Context Copy link
I think it's reasonable to point to those as bounds on "intelligence" (which is itself poorly defined): if your super intelligence model is effectively "omniscient", that part is provably not possible. What is possible seems a bit unclear, but I won't be surprised if it's "less than Skynet" even if it still changes nearly everything.
Yeah, some people seem to think that a smart human can, in general, analyze a computer program and reliably determine if it will halt or not. I'm almost certain that this is not the case.
I think people are used to looking at computer programs which are well organized and therefore it's possible to construct an accurate model of the program and use that model as a shortcut in determining whether the program will halt. So for example, you might study a program; realize that it outputs all prime numbers from 2 on up, and use your knowledge that there is an infinite number of primes to conclude that the program will not halt.
Yeah, I think these arguments are pure cope. Even if there is something humans can do which AI fundamentally cannot do (doubtful) the technology will almost certainly be wildly transformative.
It's trivially easy to show that you can't.
If you can tell whether this program will halt or not, Goldbach would like a word with you.
Very good point, thank you for posting this.
More options
Context Copy link
More options
Context Copy link
All the subjective things? Enjoy a beautiful sunrise? Love another as a spouse? Raise a child? Be a friend? Believe in God? Meaningful, not just aesthetic art?
Beyond that, all existent AI I'm aware of was trained on existing human knowledge. Without a human-derived training corpus, what is it capable of?
Not saying those could never happen, but it's not obvious to me that they could.
Well I think it depends on how you define the word "enjoy." In any event, in evaluating AI's capabilities, I think it's best to focus on inputs and outputs. There's really no way to objectively verify an AI's subjective feelings (actually there's no way to objectively verify other people's subjective feelings.)
I'm pretty confident that AI will absolutely be able to do all the tasks involved in these things. Unless of course you want to define these activities so as to require a human element.
Without a human-derived training corpus, what is any of us capable of?
I'm not convinced the human element definition is wrong, even if I wouldn't fully endorse it today either.
Demonstrably, together we got to where we are now. It'd be interesting to see how well AI could start from scratch.
Ok, well suppose someone wants to go to a bar and have their drink served by a real human bartender. By definition, that's not a task which can be replaced by AI.
I would guess that if you gave it hundreds of millions of years, it could do pretty well. I should probably note that this issue is kind of academic. I mean, if we stipulate for the sake of argument that development of AI is dependent on human input, it doesn't really say much about what AI could or couldn't do in the future -- standing on the shoulders of giants.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
You can always counter that Heisenberg's uncertainty principle pretty much guarantees that AI will emerge. [1]
[1] I know it is not true. But disproving it requires to actually think for a moment. Or at least ask ChatGPT to disprove it for you.
How does Heisenberg's uncertainty have anything to do with AI?
In the same way the Halting Problem has.
My idea is that it is such a nonsense that just figuring out what the fuck does it mean requires substantial use of one's brain. And once your opponent starts using its brain, there is small chance that the inertia will carry to examining their own beliefs and arguments.
More options
Context Copy link
Someone introduced a pretty powerful blunt to this roundtable.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link