This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
The singularity is coming in 2027 and no one cares.
Dwarkesh is out with a new essay in which he argues that there is a high chance (not 100%, but like 75%) that we get an intelligence explosion in the next 12 months equal in magnitude to the jump between GPT-3.5 and GPT-6.
This jump would be caused by the automation of AI research.
I highly, highly recommend giving this a read or a listen. Dwarkesh is, in my mind, the most thoughtful public intellectual around today (sorry Scott) and is known for his in depth research and thoughtful interviews. In his essay, it's likely that he addresses whatever objections that you have that just sprung to mind, so just give it a read. Really.
At this point, it seems the only thing that would stop superintelligence from forming is an unknown unknown, as in there is some fundamental constraint or that hasn't been identified, or an emergent phenomenon that only reveals itself in the future. I guess this has all been rehashed a million times, and I don't have a lot to add beyond what Dwarkesh said, so I'll revert to the purpose of this board which is to make some social commentary.
People are completely unprepared for what's coming.
In financial forums that I visit, and in youtube videos I watch, people are entirely dismissive of what is happening at OpenAI and Anthropic. "It doesn't really think! They're losing money! Drug development, no way! It only works for software! The software is buggy! They'll be bankrupt in a year! Switching costs are zero! Cheap Chinese models will win!".
Etc.. Etc... Etc...
Some concepts are just too high IQ for most people. Probably most people with an IQ below 100 don't really understand supply and demand for example. And the floor for understanding recursive self improvement is a lot higher.
Everything feels pretty normal now. But there's at some point going to be a big oh shit moment when social consensus hits and 90% of the population goes from denialism to doomerism in a short period. I have a hard time predicting what will happen then except that I'd assign a 50% chance that Anthropic will be the most valuable company on Earth in 12 months. Cold comfort. Superintelligence is coming and it's going to change everything. Maybe it will be good...
This isn't my read on things. Average and dumb people can understand the concept just fine. They might not like it and the changes it brings, but they absolutely recognize the threat; it's been communicated by, if nothing else, pretty simple and direct sci fi stories that they've seen on the big screen.
But it takes a higher IQ person to come up with superficially insightful but wrong/irrelevant points that leads to true AI denialism (as opposed to AI opposition). Stochastic parrots, Chinese rooms, explanations of why a trend has to stop. They've heard of P =! NP; surely that prevents AI! They can construct a TESCREAL genealogy of eugenics to AI, therefore it's bad and fake. Etc.
There are stronger arguments, of course, but I'd classify questions around things like model performance generalizing beyond RLVR friendly tasks as a different category than true denialism.
Right - I think the OP comes off a bit too much like "most people aren't as smart as me", whereas I suspect that most people understand the central plot point of The Terminator.
"We build a superintelligent computer and things go badly" isn't some sort of weird esoterica. It's been a theme in popular media for over half a century. You don't need a high IQ to understand, "Maybe we build a computer and the computer improves itself and then the computer destroys us".
Charitably, maybe the OP means that most people don't think that's likely in reality, or that most people don't understand the specific mechanism that he posits for it. I think the former is probably true and the latter is probably false. I think most people don't expect Skynet to appear, but most people probably can understand the concept of a computer that makes itself smarter. It's really not a very complicated idea.
It's weird that it doesn't really figure into popular ideas of AI, though, isn't it?
In elite non-fiction we get "Speculations Concerning the First Ultraintelligent Machine" in 1965.
In popular fiction, just looking at your linked comment: you're right that "AI might be dangerous" is well-attested in popular fiction, but "AI might upgrade its intellectual capabilities extremely rapidly" not so much. WOPR/Joshua and HAL are dangerous mostly because of how smart they aren't, and Skynet and AM gain self-awareness but don't obviously get much smarter. There's a TNG episode where Barclay both gets much smarter and integrates with the computer, but it's all external influence, no runaway positive-feedback loop involved. Data is a little more realistic than C3-PO (the droid who's basically sapient but who isn't even obsolete decades later, much less creating his own successors), but it's still not surprising when Lal doesn't work and his research stops there.
The earliest popular fiction I know of featuring recursive self-improvement is A Fire Upon the Deep, so of course I went to the TVTropes page to see what the trope name was and if the trope page listed anything earlier ... and the only trope I see for the Blight is "Eldritch Abomination", nothing more specific. Even in that case, the self-improvement is essentially a "reboot" of previously-existing capabilities; in a sense the main interaction of the story with the idea of recursive self-improvement is that the "zones of thought" are a necessary gimmick to create a plausible future setting where superintelligence is a big deal but RSI isn't.
"AI capability is a function of accumulated research, and if that function's growth rate isn't slow enough then after AI can do any AI research y'=f(y) blows up" ... simple enough to squeeze into two clauses (for anyone who's had Calc 1; for "most people" maybe add a second or third sentence too), sure. But if you then add "we don't have reason to believe its growth rate will be slow before it hits far-superhuman levels" and "artificial-intelligence research is probably easier to do right than artificial-ethics research" then this has obvious implications of the Skynet variety, yet it's surprisingly hard to find people who were taking this seriously who didn't get there via Bostrom and Yudkowsky.
Her involves the AIs explicitly engaging in self improvement until they ultimately decide to go rogue and abandon humanity (a relatively optimistic outcome).
More options
Context Copy link
There are some things that come close. Like Arnie's explanation of Skynet's history in Terminator 2, "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 AM Eastern Time, August 29th."
There is also the TNG episode where in response to being asked to create a holodeck program capable of posing a challenge to Data's intelligence, the Enterprise's computer creates a holodeck program, Professor Moriarty, that is so smart that the crew only barely manage to defeat it. It's not explained whether the ship's computer was always that smart, or whether it did something novel in response to the prompt. Still suggestive, though.
I don't think it's really explored in the series, but the difference between Data, a self-aware android and character, and the ship's computer, which seems on par with Claude (or so) feels particularly salient. Maybe at the time it was a convenient tool for the writers, but I'm not sure how I'd describe it even. For lack of a better term, Data is shown to have a soul, but the computer does not. All sides seem to have lots of strong opinions on which our current models more strongly resemble.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link