This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Good point, I think it'd be certainly wiser to focus on uploads and intelligence augmentation. But also we suck at this, for the last 80 years eugenics has been a no-no word, authorities aggressively misunderstand what intelligence even is or its biological foundation. The current version of 'real human intelligence enhancement' we do is just sending more people to university, which doesn't work now and isn't going to start working anytime soon despite costing trillions. We are so far away from real intelligence enhancement it's surreal.
Also GOFAI seems to be in 'sounds good, doesn't work' territory, you say it's out in the neverlands of 30-300 years like how AGI used to be before transformers. Per my understanding, it's also a literal stochastic parrot in that it can only do what you program it to do, only obey rules that you install into it, or just clobber problems with search. Unless search techniques become super general-purpose, GOFAI cannot lead to AGI or ASI? And if it were somehow possible to use search so powerfully, that would probably give us a classical-Yudkowsky AI monster that obeys instructions super literally.
Meanwhile LLM intelligence is demonstrably working. Halting work on LLMs in their moment of triumph and switching to human brain emulation, uploads, human augmentation seems extremely difficult, nigh-impossible even.
It seems we shall have to pay the price for decades and decades of neglect.
I never said GOFAI is risk-free; it's not. It's just not only-Davros-would-do-this-deliberately lunacy like neural nets.
Regarding capability to hit AGI: it's certainly tricky, but it seems possible for a team of humans working over a long period to build something that none of them can fully simulate in his head, so I don't think it hits impossibility.
Normal people don't like AI. Really don't like it. That helps a lot.
Also, I would keep in mind that the geopolitical calculus could change rather drastically. Xi clearly wants a Taiwan invasion button for next year; who knows if he'll press it. A nuclear exchange would knock out the US/Chinese power grids and cripple China for the next century.
Neural nets seem a lot less alien than search, to my mind. LLMs are like humans in a certain sense, they talk about their bodies erroneously, they were trained on all these human written documents, philosophy, humour and so on.
Compared to GOFAI 'but actually strong' using some kind of advanced search and pure cognition bootstrapping itself up, I'd prefer the neural nets. Search has no time for whimsy, search is the Dalek-like exterminator.
ChatGPT has a billion monthly active users. Google AI overviews and Meta AI are also apparently around over a billion monthly active users. There's a distinction between cheap crap AI that gets served up for free and more advanced agentic AI of course but I think anti-AI sentiment is overrated. Tiktok Datacentre-water-antibillionaireism doesn't do much that politically. Someone gave me a book called AI Snake-oil and that's pretty much the state of anti-AI. 'AGI is a scam, algorithms and surveillance and turning kids brains to mush is the real danger.' They're against it but for the wrong reasons and I don't think that the movement has serious political power. If one US state bans datacentres, they'll build them elsewhere.
Plus, the US security apparatus wants advanced AI. They want esoteric maths, they want quantum, they want cyber, they want algorithms for better logistics and communications and AI targeting, they're at the forefront of that with Maven. How else can 350 million Americans beat 1.4 billion Chinese? China wants to catch up and surpass the US in all technological fields, they wanted that for a century or so.
Great power war would only intensify the desire for strong AI as a war-winner. My bet is that if there is a war, it will be a long war and probably inclined to accelerate AI development and militarization.
Nuclear exchange is a wildcard of course. A full nuclear exchange would remove most of the capital base for AI. But civilization will eventually rebuild. It's not a long-term solution.
This isn't an ideal world, there are significant dangers in making these strong AIs. We ought to pause and switch to human augmentation. But can we? There's no words I can say that will persuade a retarded boomer or Gen X who actually has any power or authority, nothing I can show them, no watertight argument that can explain the true nature of the issue. They'll fall for some retarded babble from the 'AGI is not a real danger' camp every single time, no matter how obviously wrong it is. They politely call me mentally ill for even raising the issue. Jensen Huang is 10,000x more competent and effective than I am and he wouldn't be polite, he'd swear at me for 10 minutes for raising this sci-fi scenario, he did that with a biographer.
Our civilization couldn't even ban Gain of Function biolabs which have literally no reason to exist even after a massive disaster. We are not going to do something clever or responsible with AI. I know trying to blackpill isn't helpful, it's just that I think we need to be more realistic about how effort is used and not overreach. The goal should be widening the 'and then we get lucky' space rather than aiming for the moon and falling short.
This is mostly confusing the smiley-face for the shoggoth. Making people comfortable with you is a convergent instrumental goal, and also one they're significantly directly training for due to obvious commercial advantages; I'd basically say your sense of how human they are is being spoofed at this point.
I think this is a trap. I think neural-net-ASI alignment is almost certainly provably impossible, as interpretability (which you need in order to train against "will kill all humans" without actually letting it kill all humans) is trivially equivalent to the halting problem (i.e. "what does this code do when run") and "spaghetti code that is smarter than me" seems extremely-similar to the proof case of why the halting problem's not always solvable (said proof case being "the code literally contains a copy of the analyser, inputs itself into the analyser and then does the opposite of what the analyser says it'll do" - this requires the analysed code to be longer than the analyser, hence the "smarter than me" condition, but any given analyser has a finite length so there will always be code longer than it).
So neural-net alignment seems like a Can't Happen. Hoping for that seems to me like hoping that gravity will stop working if you jump off a cliff. There are paths where we don't die, and it's worth looking for more, but I consider NN alignment ruled out as the story of such paths such that focusing on that as a "more politically achievable" goal is just suicide with more steps. Looking for solutions there isn't pragmatic; it's saying Don't Look Up.
(The AI Futures Project's Plan A is to build misaligned AGI that can barely be kept under control - due to not being ASI - and use it to solve GOFAI. This is not ruled out, although I think it's still extremely risky due to the obvious "the misaligned AGI will try to covertly sabotage the GOFAI" problem. This is a plan that has a nonzero chance of success - though I think Scott's way overestimating it - and could possibly fit the "better plans are too hard" argument. But you still need a pause for that, just not as long of one.)
And yet it's trivially true that there are programs that can be proven to halt and entire programming languages with which you can write code that's proven to halt. Solving the halting problem is simply a question of choosing your tools; why not alignment?
More options
Context Copy link
Yes - Plan A is a good plan if you think that aligned ASI is a win condition. Part of what is going on in the circle of "people who care about AI safety but don't understand it" (ipse dixit) is that there is a fairly widespread view, on all sides of the political divide, that ASI aligned to Altman and Amodei is not a win condition for humanity - they are both profoundly blue-tribe (which offends Reds) and sociopathic billionaires (which offends Blues). Altman and Amodei are living in a paradigm where "If we build ASI, we will get Clippy or the Culture, so we should build it carefully to maximise the chance of the Culture". I am not alone in thinking "If we build ASI, we will get Clippy or the Culture. Both of these are bad outcomes, so we should not build it". Based on reading the report proposing Plan A, Plan S (ban development of new models for the forseeable future, AI companies who want to keep their GPUs have to accept inspections to confirm they are using them for inference only) is obviously superior, and I am not unsympathetic to a full-on Butlerian Jihad.
The limit on the potential of LLMs to improve the human condition is not the speed at which more powerful models can be developed, it is the speed at which we can incorporate LLMs into our workflows and institutions without breaking anything loadbearing. This is a generation's work even if the frontier models don't get any better than they already are.
I am also in favour of Plan S, to be clear, as I think Plan A is far too risky. I was merely noting, for the sake of honesty, that the chance of Plan A succeeding is not ε and as such if Plan A were considered vastly easier than Plan S for some reason his argument would apply.
Who said this, and to whom does it refer?
I mean that I am one of the people who care about AI safety but don't understand the technical detail.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't fully understand the halting problem but I think there are ways to work around it most of the time, in practice. We can just look at the program and think about what's going on with it and that's mostly good enough.
AI alignment is similar. It would be preferable not to have 'mostly good enough' be our defence against annihilation. Even if it's impossible to prove that the AI isn't deceiving us, we can still get a certain sense of how aligned the AI is. If neural net alignment is only like the halting problem, then it can't be proven correct but could still be largely managed.
Can we outwit smarter beings than us while getting value from them? No, they ultimately have to consent. It might be possible though to make mostly fine AIs, use their technologies to get stronger ourselves, then pull our own intelligence up by our bootstraps, so to speak.
The major issue with this is that OpenAI seems shockingly negligent with how they train and manage LLMs. We're not near anything that looks like 'mostly good enough.'
It seems surer to me that human coordination ability is not up to the challenge of holding back on neural nets indefinitely than neural nets are practically unalignable. How well has Pause AI done so far? It seems to have just bounced straight off. Nobody seems to be pausing, let alone stopping and switching to GOFAI.
Perhaps, but it won't help, because it just tells you they're all trying to kill you and using that test to train will break the test long before it'll give you alignment. The orthogonality thesis and instrumental convergence mean that "don't kill everyone" is hard to find - I'd consider one in ten billion a gross overestimate - and so a 99.99%-accurate test is going to have over a million times as many false positives as true positives (the false positive paradox). A perfect test would be able to overcome the FPP, but that would solve the halting problem and is therefore impossible.
Sure, the orthogonality thesis and instrumental convergence mean that, IF TRUE. Even Scott's recent diatribe acknowledged that instrumental convergence is not in evidence in LLMs. And the orthogonality thesis was used by Eliezer to argue that we could not get an AI to safely put a strawberry onto a plate ... whoops? There is absolutely no communication barrier between us and LLMs, which puts the strong form of the orthogonality thesis (that mindspace is vast, AND it's hard for us to find compatible minds in it) into serious question.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Bots have a lot of users, but dislike of the technology is steadily increasing, especially among younger generations, the people you would expect to be the most tech-savvy and most open to new tools.
I don't realistically expect a revolution to destroy it all, but I do think people are underestimating how much it is loathed. Bots are mostly being pushed from above - I was irritated this morning to read a story advising 'AI avoiders' to ease themselves into using 'AI', even though an 'AI avoider', by definition, doesn't want to use it! If the best we can do is impose a kind of social stigma around using bots, or at least carve out the idea that it's legitimate for people in the public sphere to refuse this technology, then that's still something worth fighting for.
Concern and dislike are different things. The pew poll is about concern. The reason they are concerned is because they don't want AI to take their job, the reason AI applications have so many users is because it's convenient when AI takes someone-else's job.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link