This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I just got done listening to Eliezer Yudkowski on EconTalk (https://www.econtalk.org/eliezer-yudkowsky-on-the-dangers-of-ai/).
I say this as someone who's mostly convinced of Big Yud's doomerism: Good lord, what a train wreck of a conversation. I'll save you the bother of listening to it -- Russ Roberts starts by asking a fairly softball question of (paraphrasing) "Why do you think the AIs will kill all of humanity?" And Yudkowski responds by asking Roberts "Explain why you think they won't, and I'll poke your argument until it falls apart." Russ didn't really give strong arguments, and the rest of the interview repeated this pattern a couple times. THIS IS NOT THE WAY HUMANS HAVE CONVERSATIONS! Your goal was not logically demolish Russ Roberts' faulty thinking, but to use Roberts as a sounding board to get your ideas to his huge audience, and you completely failed. Roberts wasn't convinced by the end, and I'm sure EY came off as a crank to anyone who was new to him.
I hope EY lurks here, or maybe someone close to him does. Here's my advice: if you want to convince people who are not already steeped in your philosophy you need to have a short explanation of your thesis that you can rattle off in about 5 minutes that doesn't use any jargon the median congresscritter doesn't already know. You should workshop it on people who don't know who you are, don't know any math or computer programming and who haven't read the Sequences, and when the next podcast host asks you why AIs will kill us all, you should be able to give a tight, logical-ish argument that gets the conversation going in a way that an audience can find interesting. 5 minutes can't cover everything so different people will poke and prod your argument in various ways, and that's when you fill in the gaps and poke holes in their thinking, something you did to great effect with Dwarkesh Patel (https://youtube.com/watch?v=41SUp-TRVlg&pp=ygUJeXVka293c2tp). That was a much better interview, mostly because Patel came in with much more knowledge and asked much better questions. I know you're probably tired of going over the same points ad nauseam, but every host will have audience members who've never heard of you or your jargon, and you have about 5 minutes to hold their interest or they'll press "next".
Yud is not trying to sway honest-to-God normies with this podcast tour (and people who've greenlit this multipronged astroturf of AI doom don't expect him to either, but that's a conspiratorial aside). He never could be popular among normies, he never will, he's smart enough to realize this. His immediate target is… well, basically, nerds (and midwitted pop-sci consumers who identify as nerds). Nerds in fact appreciate intellectual aggression and domination, assign negligible or negative weight to normie priors like «looks unhinged, might be a crackpot», and nowadays have decent economic power, indeed even political power in matters pertaining to AI progress. Nerds are not to be underestimated. When they get serious about something, they can keep at it for decades; even an autist entirely lacking in affective empathy and theory of mind can studiously derive effective arguments over much trial and error, and a rationalist can collate those tricks and organize an effective training/indoctrination environment. Nerds will get agitated and fanatical, harangue people close to them with AI doom concerns, people will fold in the face of such brazen and ostensibly well-informed confidence, and the future will become more aligned with preferences of doomers. Or so the thinking goes. I am describing the explicit logic that's become mainstream in one rat-adjacent community I monitor; they've gotten to the stage of debating pipelines for future AI-Alignment-related employment, so that the grift would never stop.
But on the object level:
You assume there is a minimal viable payload that he has delivered to you and others, and which is viable without all that largely counterproductive infrastructure. That is not clear to me. Indeed, I believe that the whole of Yud's argument is a single nonrobust just-so narrative he's condensed from science fiction in his dad's library, a fancy plot. It flows well, but it can be easily interrupted with many critical questions. He describes why timelines will «converge» on this plot, the nigh-inevitability of that convergence being the central argument for urgency of shutting down AI, but its own support is also made up of just-so stories and even explicit anti-empiricism; and once you go so deeply you see the opposite of a trustworthy model – basically just overconfident logorrhea.
That's exactly why Yud had to spend so many years building up the delivery vehicle, an entire Grand Theory, an epistemological-moral-political doctrine, and cultivating people who take its premises on faith, who all use the same metaphors, adhere to the same implicit protocol. His success to date rests entirely on that which you're telling him to drop.
Here's how Zvi Moskowitz understands the purpose of Yud's output, «its primary function is training data to use to produce an Inner Eliezer that has access to the core thing». (Anna Salomon at CFAR seems to understand and apply the same basic technique even more bluntly: «implanting an engine of desperation» within people who are being «debugged»).
In a sense, the esotericism of Yuddite doctrine is only useful, it had insulated people from pushback until they became rigid in their beliefs. Now, when you point at weak parts in the plotline, they answer with prefab plot twists or just stare blankly, instead of wondering whether they've been had.
Nerdy sects work; Marxism only the bloodiest testament to this fact. Doomsday narratives work too, for their target audience (by the way, consider the similarity of UK's Extinction Rebellion and Yuddites' new branding «Ainotkilleveryoneism»). They don't need to work by being directly compelling to the broader audience or by having anything to do with truth.
P.S. Recently I've encountered this interesting text from an exactly such an AI-risk-preoccupied nerd as I describe above: The No-Nonsense Guide to Winning At Social Skills As An Autistic Person
His social well-being is now literally predicated on his investment in EA-AI stuff, so I'd imagine he goes far, and this easily counts more for Yud's cause than 10k positive comments under another podcast.
If it makes a difference, I recently updated away from a P(doom) of ~70% to a mere 40ish recently.
This was on the basis of empirical AI research contradicting Yud's original claims that the first AGI would be truly alien, drawn nigh at random from the vast space of All Possible Minds.
As someone on LW put it, which struck the important epiphany for me, was that LLMs can be distilled to act identically to other LLMs by virtue of training on their output.
And what do you get if you distill LLMs on human cognition and thoughts (the internet)? You get something that thinks remarkably like us, despite running on very different hardware and based off different underlying architecture.
Just the fact that LLMs have proven so tractable is cause for modest optimism that we'll wrangle them yet, especially if the superhuman models can be wrangled through RLHF to be robust to assholes commanding them to produce or execute plans to end the world.
Of course, it's hard to blame Yud for being wrong when, when written, everyone else had ideas that were just as widely off the mark as he was.
No it isn't. When you are speculating wildly on what might happen, you rightly bear the blame if you were way off the mark. If Yud wasn't a modern day Chicken Little, but was just having some fun speculating on the shape AI might take, that would be fine. But he chose to be a doomer, and he deserves every bit of criticism he gets for his mistaken predictions.
Mostly disagree - speculation should be on the mark sometimes, but being correct 1/50th of the time about something most people are 0% correct about (or even 1/50th correct about, but a different 50th) can be very useful. If you realize the incoherence of Christianity and move to Deism ... you're still very wrong, but are closer. Early set theories were inconsistent or not powerful enough, but that doesn't mean their creators were crackpots. Zermelo set theory not being quite right didn't mean we should throw it out!. This is a different way of putting scott's rule genius in, not out. And above takes aren't really 'Yud made good points but mixed them with bad ones'
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link