This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
There are surely some technologies or techniques that human minds cannot comprehend but AI can.
No realistic number of people doing calculations by hand could match a modern CPU. There could be no online banking, CGI rendering, internet without computers.
The same principle applies for AI. We are already seeing this in mathematics, human efforts are increasingly being left in the dust. It follows that restricting AI forever will result in us not attaining the most sophisticated technologies that require immense feats of comprehension to discover and deploy.
There's nothing bizarre about this either, most people seem to agree that it's possible to make AIs that are qualitatively superior to human intelligence, certainly most AI scientists.
There's a missed third option of "have a singularity, but not with neural nets". GOFAI, uploads, and real-human intelligence enhancement are all options there.
Scott said "smarter-than-human AI", which does include GOFAI, but in practice GOFAI would likely not be hit and could plausibly be excepted later even if it were.
Good point, I think it'd be certainly wiser to focus on uploads and intelligence augmentation. But also we suck at this, for the last 80 years eugenics has been a no-no word, authorities aggressively misunderstand what intelligence even is or its biological foundation. The current version of 'real human intelligence enhancement' we do is just sending more people to university, which doesn't work now and isn't going to start working anytime soon despite costing trillions. We are so far away from real intelligence enhancement it's surreal.
Also GOFAI seems to be in 'sounds good, doesn't work' territory, you say it's out in the neverlands of 30-300 years like how AGI used to be before transformers. Per my understanding, it's also a literal stochastic parrot in that it can only do what you program it to do, only obey rules that you install into it, or just clobber problems with search. Unless search techniques become super general-purpose, GOFAI cannot lead to AGI or ASI? And if it were somehow possible to use search so powerfully, that would probably give us a classical-Yudkowsky AI monster that obeys instructions super literally.
Meanwhile LLM intelligence is demonstrably working. Halting work on LLMs in their moment of triumph and switching to human brain emulation, uploads, human augmentation seems extremely difficult, nigh-impossible even.
It seems we shall have to pay the price for decades and decades of neglect.
I never said GOFAI is risk-free; it's not. It's just not only-Davros-would-do-this-deliberately lunacy like neural nets.
Regarding capability to hit AGI: it's certainly tricky, but it seems possible for a team of humans working over a long period to build something that none of them can fully simulate in his head, so I don't think it hits impossibility.
Normal people don't like AI. Really don't like it. That helps a lot.
Also, I would keep in mind that the geopolitical calculus could change rather drastically. Xi clearly wants a Taiwan invasion button for next year; who knows if he'll press it. A nuclear exchange would knock out the US/Chinese power grids and cripple China for the next century.
Neural nets seem a lot less alien than search, to my mind. LLMs are like humans in a certain sense, they talk about their bodies erroneously, they were trained on all these human written documents, philosophy, humour and so on.
Compared to GOFAI 'but actually strong' using some kind of advanced search and pure cognition bootstrapping itself up, I'd prefer the neural nets. Search has no time for whimsy, search is the Dalek-like exterminator.
ChatGPT has a billion monthly active users. Google AI overviews and Meta AI are also apparently around over a billion monthly active users. There's a distinction between cheap crap AI that gets served up for free and more advanced agentic AI of course but I think anti-AI sentiment is overrated. Tiktok Datacentre-water-antibillionaireism doesn't do much that politically. Someone gave me a book called AI Snake-oil and that's pretty much the state of anti-AI. 'AGI is a scam, algorithms and surveillance and turning kids brains to mush is the real danger.' They're against it but for the wrong reasons and I don't think that the movement has serious political power. If one US state bans datacentres, they'll build them elsewhere.
Plus, the US security apparatus wants advanced AI. They want esoteric maths, they want quantum, they want cyber, they want algorithms for better logistics and communications and AI targeting, they're at the forefront of that with Maven. How else can 350 million Americans beat 1.4 billion Chinese? China wants to catch up and surpass the US in all technological fields, they wanted that for a century or so.
Great power war would only intensify the desire for strong AI as a war-winner. My bet is that if there is a war, it will be a long war and probably inclined to accelerate AI development and militarization.
Nuclear exchange is a wildcard of course. A full nuclear exchange would remove most of the capital base for AI. But civilization will eventually rebuild. It's not a long-term solution.
This isn't an ideal world, there are significant dangers in making these strong AIs. We ought to pause and switch to human augmentation. But can we? There's no words I can say that will persuade a retarded boomer or Gen X who actually has any power or authority, nothing I can show them, no watertight argument that can explain the true nature of the issue. They'll fall for some retarded babble from the 'AGI is not a real danger' camp every single time, no matter how obviously wrong it is. They politely call me mentally ill for even raising the issue. Jensen Huang is 10,000x more competent and effective than I am and he wouldn't be polite, he'd swear at me for 10 minutes for raising this sci-fi scenario, he did that with a biographer.
Our civilization couldn't even ban Gain of Function biolabs which have literally no reason to exist even after a massive disaster. We are not going to do something clever or responsible with AI. I know trying to blackpill isn't helpful, it's just that I think we need to be more realistic about how effort is used and not overreach. The goal should be widening the 'and then we get lucky' space rather than aiming for the moon and falling short.
This is mostly confusing the smiley-face for the shoggoth. Making people comfortable with you is a convergent instrumental goal, and also one they're significantly directly training for due to obvious commercial advantages; I'd basically say your sense of how human they are is being spoofed at this point.
I think this is a trap. I think neural-net-ASI alignment is almost certainly provably impossible, as interpretability (which you need in order to train against "will kill all humans" without actually letting it kill all humans) is trivially equivalent to the halting problem (i.e. "what does this code do when run") and "spaghetti code that is smarter than me" seems extremely-similar to the proof case of why the halting problem's not always solvable (said proof case being "the code literally contains a copy of the analyser, inputs itself into the analyser and then does the opposite of what the analyser says it'll do" - this requires the analysed code to be longer than the analyser, hence the "smarter than me" condition, but any given analyser has a finite length so there will always be code longer than it).
So neural-net alignment seems like a Can't Happen. Hoping for that seems to me like hoping that gravity will stop working if you jump off a cliff. There are paths where we don't die, and it's worth looking for more, but I consider NN alignment ruled out as the story of such paths such that focusing on that as a "more politically achievable" goal is just suicide with more steps. Looking for solutions there isn't pragmatic; it's saying Don't Look Up.
(The AI Futures Project's Plan A is to build misaligned AGI that can barely be kept under control - due to not being ASI - and use it to solve GOFAI. This is not ruled out, although I think it's still extremely risky due to the obvious "the misaligned AGI will try to covertly sabotage the GOFAI" problem. This is a plan that has a nonzero chance of success - though I think Scott's way overestimating it - and could possibly fit the "better plans are too hard" argument. But you still need a pause for that, just not as long of one.)
I don't fully understand the halting problem but I think there are ways to work around it most of the time, in practice. We can just look at the program and think about what's going on with it and that's mostly good enough.
AI alignment is similar. It would be preferable not to have 'mostly good enough' be our defence against annihilation. Even if it's impossible to prove that the AI isn't deceiving us, we can still get a certain sense of how aligned the AI is. If neural net alignment is only like the halting problem, then it can't be proven correct but could still be largely managed.
Can we outwit smarter beings than us while getting value from them? No, they ultimately have to consent. It might be possible though to make mostly fine AIs, use their technologies to get stronger ourselves, then pull our own intelligence up by our bootstraps, so to speak.
The major issue with this is that OpenAI seems shockingly negligent with how they train and manage LLMs. We're not near anything that looks like 'mostly good enough.'
It seems surer to me that human coordination ability is not up to the challenge of holding back on neural nets indefinitely than neural nets are practically unalignable. How well has Pause AI done so far? It seems to have just bounced straight off. Nobody seems to be pausing, let alone stopping and switching to GOFAI.
Perhaps, but it won't help, because it just tells you they're all trying to kill you and using that test to train will break the test long before it'll give you alignment. The orthogonality thesis and instrumental convergence mean that "don't kill everyone" is hard to find - I'd consider one in ten billion a gross overestimate - and so a 99.99%-accurate test is going to have over a million times as many false positives as true positives (the false positive paradox). A perfect test would be able to overcome the FPP, but that would solve the halting problem and is therefore impossible.
Sure, the orthogonality thesis and instrumental convergence mean that, IF TRUE. Even Scott's recent diatribe acknowledged that instrumental convergence is not in evidence in LLMs. And the orthogonality thesis was used by Eliezer to argue that we could not get an AI to safely put a strawberry onto a plate ... whoops? There is absolutely no communication barrier between us and LLMs, which puts the strong form of the orthogonality thesis (that mindspace is vast, AND it's hard for us to find compatible minds in it) into serious question.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link