This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I suppose you're correct, although it is my impression that ~everyone who believes in short AI timelines / AI safety regulation now believes in scaling compute being the way to AGI. Even non-LLM believers are primarily looking for new ways to apply the bitter lesson, and the focus of AI safety regulation is on trying to control world compute, which would be completely pointless if Gary Marcus got the final laugh vis a vis symbolic AI.
Yeah, this is my point. Empirical evidence is that the sum total of AI safety's achievements is, against their own goals, to accelerate the founding of OpenAI/Anthropic and the AI race dynamic, while having made no progress on their actual goals (not even the authors expect Plan A to realistically happen).
Of course, it's possible they could turn it around and heroically pull humanity into post-scarcity abundance, the same way that Marxism still has time to eventuate the communist revolution (arguably it's kind of sort of happening with the DSA); but if the stakes are infinite, then there's nothing to do but keep going, even if you're going in the wrong direction, which is why getting Pascal's Mugged is so dangerous.
This is the point I'm trying to make though; perhaps it was poorly communicated. At the turn of the 20th century, Marxism was in vogue amongst many Western intellectuals, because we had not yet seen what would happen with communism in action.
Right now, we are at the equivalent stage of AI safety, where it is in vogue in certain intellectual circles; we have not yet seen what will happen with AI safety, but it shares many structural similarities to the Marxist movements a hundred years ago.
After the transformers paper came out it was only a matter of time, this way at least the labs are full of safety people who might be able to prioritize alignment work. The alternative was optimistically we get a year or so more time but no influence.
This prove too much. early liberal, proto-capitalist, thought also shared these structural elements.
"Might" is a weak word. Historically, "join your enemies to reform them from within" has pretty much never achieved the desired aims; it seems straightforwardly true that Moloch has won in terms of incentives to push capabilities and accelerate.
I think if you accept the premise that the world is truly doomed on such a short timescale, then having a few more years is better, actually, than trading billions of QALY's for some nebulous idea of influence that has had no empirical benefit.
Perhaps. It would be easier to engage with this if you elaborated on why you think this is the case.
Framing this as a simple enemies or allies distinction is sanding away anything useful. Like if the "enemy" was "war" then one might naively believe that creating an army is joining the "enemy". But if what is bad about war is the killing and oppression then you don't prevent that by disarming, only ensure that the people who win are less noble and less capably opposed.
It's already the case thst people accuse those like yud of being an uninvolved crank who doesn't understand the problem because he doesn't have experience building the models. Ai safety people would have less, not more influence if they abstained.
Given the infinity of total annihilation any qaly calculation is swamped by even a tiny amount of leverage on probability of averting disaster. You'd need to set the capabilities frontier back some appreciable fraction of the sun's remaining lifespan for this to be a meaningful calculation. Fortunately I think having people in leadership positions of the labs gives us more than a tiny amount of leverage.
This is of course vague because you didn't yourself outline what these structural elements are. Seems like some vague notion of the necessity of action to avoid catastrophe. If "we'll be out competed by peer nations who liberalize" is too much of a stretch then we can point to calls to shut down the border lest the nation be dissolved or any number of other calls to action. The calling up of a militia to defend from Indian attacks. The point is that it's silly to compare any movement that wants to act for any reason to communism in order to rub its nose in that feces. You'd leave yourself completely paralyzed.
The difference is that "war" is a concept and "AGI" is, at least theoretically, being physically manifested. If the enemy is "the army about to invade" then you probably don't want to join the enemy army to convince them to turn around.
This is the exact logic that I am saying leads to very bad outcomes. If you believe that the stakes are the infinity of total annihilation, that total annihilation is imminent, and that any leverage thus has infinite value, then you can justify doing anything to gain even miniscule leverage, on nothing but the strength of your own belief. The anime character draining the lives of everyone across the world because they believe that only they can bring the world to salvation is not the protagonist of the story.
I think I have outlined my case well enough.
Adherents of both ideologies believe in an imminent Event, with infinitely high stakes attached to the Event, that they are the only ones capable of making sure the Event leads to utopia, and that at least in the case of Marxism, this set of beliefs lead to the justification of many horrible acts that did not, in fact, meaningfully facilitate a better outcome of the Event. As far as I know, this description does not apply to liberal or capitalistic thought.
You're in a war naive society and someone publishes a white paper: "people can use violence to take whatever they want from unarmed populations" You want no one to be dispossessed so you can join the capabilities army explicitly organized around preventing banditry or you can abstain. If everyone like you abstains what do you think the equilibrium on banditry will be? Certainly abstention can be defended under a number of philosophies but joining is a perfectly rational decision.
I'm responding to this scenario:
The reasoning here is faulty in the way I outlined. You're critiquing the assumptions you baked into your own scenario. I would personally inject a lot more nuance instead of these stark terms but I was willing to engage in your hypothetical. My P(doom) is much closer to 20% than the 100% you apply. 20% is still huge and easily worth mitigating with some bilateral treaties. It's you lot who seem to conflate bilateral treaties with like eternal tyranny despite no such tyranny coming from previous treaties.
I mean not exactly? Communists believe capitalism is inherently unstable and will eventually give birth to communism once buried under the weight of its contradictions or whatever. But they make no real promises of imminence. For that matter AI safety people mostly don't even commit to claiming certainty that LLM architecture will even get there, only that it could and we ought be prepared in case it does.
Not really true of Marxists. As far as I can tell Marx himself estimated the stakes at finite, maybe in the ballpark of alleviating serfdom or avoid depression. nuclear proliferation is a much better comparison. The comparison falls pretty flat in general here.
I don't know, kind of if you squint? I'll point out that at least the Yud brand is very happy to keep us at the status quo rather than trying to usher in the eschaton, it's the accelerationists who are really pushing for utopia. Certainly both have a vision of the good, but this is where I think liberalism, christianity, or basically any have this quality.
What horrible acts? Establishing bilateral treaties and, in the same vein of cartoonish libertarian discourse that equates having laws at all with infinite tyranny because noncompliance eventually escalates to violence, we might have to enforce the treaties with violence? This chain is incredibly unconvincing and conflicts violently with the previous clause because it's precisely the opposite group of people who want to do these "horrible acts" as who want to usher in the utopia.
In short this whole comparison looks incredibly forced. It looks a lot like out group homogeneity bias. And I have no earthly idea what your position actually is. You seem to simultaneously want to let it rip because you oppose safety people who want to slow down, while also distrusting the people pushing it forward? What would you actually advise the people worried about ai existential risk do?
Implicit in such claims is that "we should use the capabilities army to stop anyone else from mustering an army", hence turning into the bandits that you were so afraid of in the first place.
Much of the global south would disagree with that assessment. Still, the obvious differences between HEU and compute is that there is no justifiable civilian use for HEU, and the existential risk of HEU is plainly obvious, whereas compute is dual-use with vast civilian applications, while the existential risk is speculative at best. Any serious attempt to control compute will be useless at best, and lead to eternal tyranny / WW3 at worst.
This is why I specified the early 20th century; there was a legitimate widespread belief amongst many 20th century intelligentsia that the fall of capitalism was imminent around the WW1 revolutions and the Great Depression, when it really did look like the contradictions of capitalism were coming to roost.
No, it is indeed broadly true of 20th century Marxism. Traditional Marxism considers the communist revolution to be a final, eschatological event capable of ending the class struggle, and with the end of class struggle an end to the suffering, exploitation and conflict plaguing humanity for the rest of time. Lenin and Trotsky were both adherents to this school of thought, where they believed that they were the ones at the vanguard to the most important moment in all of history, precisely because of the immensity of the stakes.
Sure, I will concede that the Yud school of thought is less about creating utopia and more about avoiding infinite dystopia, while the Scott school is talking about ushering in utopic superintelligence by 2040. The point is that AI safetyists act like Cassandra in having very strong beliefs where they are the only ones that are clearly forecasting the imminent disaster and that the rest of society is badly mistaken; being over-confident about like, who wins the World Cup has few consequences unless you gamble too much on Kalshi, but it is very dangerous when you also believe that the stakes of your inside view are functionally infinite.
Well, 20th century Marxism directly lead to the USSR and Maoist China, neither of which could have said to been optimal for human flourishing. I'm not saying AI safety has lead to anything this bad yet, but when you start talking about nuclear war being acceptable to achieve your aims, or that we need a world government to nationalize the means of compute because the alternatives are so much worse, it does make me concerned.
Preventing others from turning bandit may be something bandits do, but is not centrally the problem of banditry. The legitimate police also do this. One who fights monster should see to it that they themselves do not become monsters, but we have different words for monster fighters and monstrosities for a reason.
How curious the need for the H in that acronym, because obviously enriched uranium does indeed have civilian use. likewise safety people are happy to allow inference datacenters so long as they, like nuclear power plants, willing to register and make clear they aren't doing weapons grade enrichment/frontier model training. It's absurdly analogous.
Facts asserted not in evidence. I happen to think at best it prevents the destruction of all value in the known universe. The odds of this are of course reasonably disputed even by me, but at best? Come on.
I'm sorry, what's the pathway to ETERNAL tyranny? If you could guarantee our human institutions would endure for an eternity then that's quite the prediction!
This is a different type of end. I will again point you at the very important difference between finite and infinite. Infinite does not mean "very large". Marxists as far as I am aware considered a transition to communism inevitable. There was a concept of very bad times spent in the desert not achieving communism that they could accelerate their way through, but human annihilation was not the default path.
You've very fundamentally misunderstood Scott if this is what you take from the predictions. He's very clear about the point of these plans being to try and lay out a path for the good future while what he's worried about at the bad ends. These are compromises with reality. A desperate attempt to steer us out of race conditions that lead to hell. Scott is not an accelerationist and I believe would be very happy to press a button pausing development. He just reasonably believes no such button exists.
This is you punished yud and folks for biting a bullet that you demand they bite. It's an annoying behavior. Again like a libertarian on hearing that you want to ban child pornography demanding that you bite the bullet that you'd kill someone over it because ultimately any law is backed up by the force of the state. It's both at once a childish reduction and a refusal to engage with the on the ground reality of international treaties. Nuclear proliferation is backed up by the threat of WW3 and yet we have no run into WW3. You'd doom us into any stupid tragedy of the commons problem with this nonsense denial of the coordination mechanism that is clearly available to us.
I mean like real suggestions, not vague tone policing. I'm plenty skeptical, that's why I'm at 20% and not 100%. you're interacting with the end product of much skepticism. What this seems to cache out to is "don't believe in AI x-risk" and sorry, that isn't where the evidence points. I'm asking you to actually consider what you'd do if you thought this thing could kill us all, not what you're negotiating for given you don't believe it does.
The difference is in epistemological certainty and scope of actions. The police don't kill without a very high certainty that it is necessary, and even when they do make mistakes the scope of the mistake is that "only" individual people die. This is extremely different to AI safety policy gambling the fate of society on epistemics one or two orders of magnitude less certain than a policeman's threshold to inflict lethal violence.
I suppose it depends on how successful you think nuclear non-proliferation actually was. In my view, pretty much every serious nation-state either openly has nuclear weapons or a turn-key program for rapidly obtaining nuclear weapons if neccessary, South Africa is the only country that has ever willingly denuclearised amidst uniquely dysfunctional transition dynamics, and this is all while nuclear weapons have an extremely concrete existential risk profile, no dual-use potential, and are economically net-negative to maintain; none of which apply to compute.
To clarify, by "useless", I mean that my belief is that the chance that such a global treaty is both workable and actually achieves its stated goals in reality rounds to 0, not that a hypothetical where everyone sits in a flower circle, sings kumbaya and shares the means of compute wouldn't theoretically work.
What I take from the predictions is that he thinks the world will end in 5 years without his desired interventions, and that he thinks the world will be a post-scarcity utopia run by superintelligence within 14 years with his desired interventions. By any reasonable outside view, this is an inside view claim of enormous magnitude with highly imminent and unbounded stakes.
Fair enough, I wasn't being as precise with my language as I could have; but the amount of all value in the known universe is also finite, not infinite. Replace every mention of mine of "infinite" with "unbounded yet finite" and I think the comparison between Marxist and AI Safety eschatological stakes still holds.
Under Yud Thought the world government needs to keep bombing the data centres for an unbounded yet finite length of time, or else AGI inevitably emerges and it's all over, and under AI 2040 Thought we hand over control of the lightcone to superintelligent AI in 2040. As far as I am concerned both of these look a lot like unbounded yet finite tyranny, even if such tyrannies might be various degrees of comfortable along the way.
I do, in fact, think this thing could kill us all, as I've mentioned a few times in the original post and replies; the same way as many mundane risks could kill me at any moment, and many other existential risks could kill us all as well. As a result, I spend my time grilling and enjoying my life while the going is good, until it inevitably ends one way or another.
I do this instead of advocating for proposals, that in my estimation, are about as likely as the Communist revolution to achieve their goals or to avoid inadvertently inflicting significant harm on the rest of society, and instead of actively working to hasten the creation of the Torment Nexus.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link