site banner
PaperclipPerfector  6d ago (text post)   22508 thread views

Culture War Roundup for the week of October 5, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

2
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Scott Alexander, the guy who hosted the Culture War Thread on his blog slatestarcodex before that landed him in so much heat (from the woke left) that it was moved to reddit under the name /r/theMotte (and from there, subsequently, to themotte.org to preempt a reddit crackdown) comes usually across as a kind and considerate person to me. Charitable interpretations are very much his thing. While he has always engaged in CW topics on occasion, his forays deep into the trenches are rare (and in my impression have become rarer compared to the SSC days).

Apparently, there is a person called Steven Pinker. I have been vaguely aware that there was a book called The Better Angels of Our Nature which was probably authored by someone, but he was not someone I really had on my radar.

From the WP article, he seems solidly Grey Tribe. Believes in a computational theory of mind, Ashkenazi Intelligence Hypothesis, critical of both the woke left (cancellations) and the MAGA right (foreign student restrictions).

The main topic where he disagrees with Scott Alexander is AI x-risk.

Scott has now responded, and his reponse is a bit less than maximum charitable.

I agree that in-person debates are confrontational and bad for truth-seeking. But I didn’t propose a debate in order to seek truth. I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel.

While Scott is diligent enough with his technical points, it is clear that he is personally hurt by Pinker's decision to ridicule AI x-risk. So his post is also a broadside fired against him. He cites Garfinkel et al 2017 who "proved" that machines will never be larger than humans using common philosophical arguments for AI being harmless.

Scott points out that while Pinker is busy berating the LW crowd for distracting from the more mundane risks of AI, x-risk advocates are willing to work hand in hand with groups with a more practical focus to combat pro-AI PACs like Leading the Future.

On model welfare (which is certainly not the core topic of the x-risk advocates):

Here you are not calling this AI torture “depraved”, you’re calling it depraved to object to it. Your argument, astonishingly, is that since we can’t be certain that AIs feel pain, we must assume that they don’t. In fact, anyone who doesn’t jump on board with this assumption (you intimate) is some sort of evil tech oligarch who wants to genocide humans. [...] I have donated $100 to a model welfare charity just to clean the stain on my soul I got from reading about it.

He accuses Pinker of endorsing the "lowest quality voices", quoting Claire Lehmann:

Thinking AI might be conscious is like thinking that there are real people talking to you from inside the TV, or that there is a tiny band playing music inside the radio.
Fine for 2 year olds to believe, but beyond that, insane.

Or

Geoffrey Hinton is the Nobel Prize winning and Turing Award winning inventor of the modern artificial intelligence paradigm. Lehmann thinks the media should “stop platforming” him [talking about persuasive AI avoiding a shutdown] to devote more time to her, a culture warrior who pivoted to AI two months ago when it got popular.

It ends with Scott seeming seriously willing to duel Pinker:

If you continue to object that debate is a poor method for determining truth, then I suppose literal dueling is the only option left. As an eminent psychologist, I’m sure you’re familiar with Cucina et al (2023), which finds a correlation of .179 - .268 between logical reasoning ability and firearms proficiency - meaning that duels are truth-tracking in exactly the way you worry debates might not be.

Scott has admitted in the comments that this is already the toned-down version of the post he originally wrote. I think that while he held back verbally (nobody get's called a "Vogon spy in a skin suit" this time), he uses his craft well to show his frustration.

For what it's worth, I think that Pinker is correct that Eliezer is in fact over-confident in p(doom|default-ASI). Just like Pinker is when he declares that any fear of doom is ridiculous.

The main difference is that for practical purposes, it does not matter much if your p(doom) is 0.99 or 0.05. Both would be pressing problems whose solution is the most important thing in the world.

Eliezer was bullish on AI capabilities when that was far out of the Overton window, back when the machines could beat humans at chess at most. Steven Pinker was tweeting that "deep learning has probably peaked" in 2019.

This might be giving too much credit to Pinker, but here is Clarke's first law:

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

AI X risk is just this round of global warming catastrophism(we're going to be underwater by 2000 2012 2020[oh wait, that was a different ridiculous hysteria we'd all be better off ignoring]) because seriously, datacenters are fragile and require constant inputs from outside that we can just cut off. Like against Skynet we can take some collateral damage if we've got to. It's not worth worrying about, it's just another made up problem to justify giving ridiculous people more power. In this case, a literal sex cult.

Presumably, Skynet was indeed in the sky, i.e., space-based data centers, or the humans would have quickly just hit the data centers. Alternatively, Skynet, having the resources to build weapons and killer robots autonomously, would have also had the resources to build and fortify data centers - heck, its human designers may have even intended this.

The premise of the Terminator movie (the original, at least) is that Skynet was about to lose to a ragtag band of human rebels and needed to resort to time travel to have a chance of survival, so it seems quite likely that it didn't have space-based data centers, and was indeed quite vulnerable to humans quickly hitting the data centers. Presumably the nuclear war was enough to disrupt the humans initially, but once they formed a resistance they could start making meaningful progress? I don't think they had resources for aircraft/space launches though.

I'm not sure this is a useful model for future malicious AI, but this specific movie AI seems not to have solved that problem.

In this case, a literal sex cult.

And what's that got to do with the price of fish and chips?

Even if we concede arguendo your Views on sexual morality, "_______'s thoughts on AI risk are invalid because _______ engages in immoral sexual practises" is literally an ad hominem argument.

How about "X's thoughts on AI risk should be downgraded because they put [a certain set of sexual practices] above all reproach and thus [apocalyptic AI risk] rank less important to them than [continuing a certain set of sexual practices]."

Also depending on the exact nature of those sexual practices, it can be a pretty good argument for keeping them as far away from the levers of power as humanly possible. I think AI poses severe but not existential risks. Either way letting the EA/rat group set AI morality will be net-negative for the majority of humanity.

“…unlike the rest of humanity, who can definitely be trusted to rank apocalyptic AI risk ahead of their own sexual practices?”

Unless you can show a nexus between the practices and the handling of risk, ad hominem arguments are symmetric. Disqualify everyone who thinks more about pussy than x-risk and you’ll be left only with the scheming eunuchs.

Unless you can show a nexus between the practices and the handling of risk

Do you think a person can perfectly separate their interest in "consensual non-consent" from their interest in not being dommed by the machine god?

This is an impossible standard that nothing could satisfy and puts meaningful information about their personalities, beliefs, etc beyond reproach because (weird) sex is the last sacred thing to seculars.

Disqualify everyone who thinks more about pussy than x-risk and you’ll be left only with the scheming eunuchs.

From what I know of the rationalists I might take my chances with the scheming eunuchs. I'll accept people that think more about pussy than x-risk so long as the pussy isn't into CNC.

Really I'd just make a geographic requirement that every AI board needs to have at least one person from Middle of Nowheresville, Flyover Country with full veto capacity.

Disqualify everyone who thinks more about pussy than x-risk and you’ll be left only with the scheming eunuchs.

The eunuchs are the ones disproportionately involved in creating the AI in the first place.

The selection effect is actually really noteworthy here because all modern eunuchs are self-made. (They don't actually call themselves eunuchs these days unless maybe you're on the WPATH mailing list- but they are very womanly men all the same. Perhaps the way certain scientific types act to give birth to some massive project is uterus-envy?)

Disqualify everyone who thinks more about pussy than x-risk and you’ll be left only with the scheming eunuchs.

Sounds like a win. When have the scheming eunuchs ever led us wrong?

[apocalyptic AI risk] rank less important to them than [continuing a certain set of sexual practices]

That risks running into the Heckler's Veto: "I will ignore your warnings of potential catastrophe unless you forswear any lifestyle choices of which I disapprove; therefore, if you do not change your lifestyle to fit my demands, you rank prevention of catastrophe as less important than your lifestyle choices." Giving in to such a demand will then invite every wowser within 20,000 kilometres to make similar demands regarding whatever bee they happen to have in their bonnets, and once you've paid the Dane-Geld, you'll never be rid of the Dane.

Also depending on the exact nature of those sexual practices, it can be a pretty good argument for keeping them as far away from the levers of power as humanly possible.

It would be, if they were engaging in the sort of things described in the more lurid claims circulating among the Blue Tribe regarding Epstein's clients. When the practises in question entirely involve mutually consenting adults, whose standards for consent are probably a fair bit less flexible than the median, the argument starts to look a lot flimsier.

That risks running into the Heckler's Veto

This is true of virtually everything, is it not? So either we run the risk of the Heckler's Veto or we say that nothing outside of their words provide additional information or context for their beliefs, and I think that is an unreasonable standard that people do not hold unless they are already sympathetic to the person/group in question.

entirely involve mutually consenting adults

Red Means No gets at the bleeding edge of the limits of consent and while it's not "eating the babies" levels of evil and/or nonsense, it's sufficient for me to say "I do not trust anyone within a few degrees of contact with having meaningful influence over anything." Not to mention the rest of the history of the community in dealing, ah, less-than-effectively with charming sociopaths until some utilitarian threshold of victims gets crossed. Consent is not magic, and apparently, it can be retroactively withdrawn anyways under current mores.

Aella has probably included "would you eat a baby for reason X" on a survey or camp application somewhere to select for sufficiently open-minded and edgy people so maybe that's not the best example. (This is a semi-joking exaggeration based on her Plzdontkillus creator camp that asked "if you had to have sex with a cow, would you rather it be alive or dead?" and her general history of asking sophomoric shock questions for... entertainment? Community filtering? etc)

I will ignore your warnings of potential catastrophe unless you forswear any lifestyle choices of which I disapprove; therefore, if you do not change your lifestyle to fit my demands, you rank prevention of catastrophe as less important than your lifestyle choices.

Funny, this is exactly how I feel about global warming 'enthusiasts'.

Of course, the difference between this and that is that said 'enthusiasts' have an obviously massive amount to gain politically from most human activity being banned like they want; the 'ban AI' faction isn't quite there yet (and dependent on who it takes down first- maybe they're on the side of weird sex, maybe they're not) so it's not clear if the origins of the group will be a liability.

When the practises in question entirely involve mutually consenting adults, whose standards for consent are probably a fair bit less flexible than the median

The usual playbook here is "change the rules, accuse of rape ex post facto", so the truth's not going to matter anyway. Of course, the weird sex people (or at least, their allies) are the same ones pushing in the #fightFor25 direction more generally, so maybe they don't have to worry about that for the foreseeable future.

The problem here is that some of your interlocutors might be actual Christians, who would really rather die (along with everyone else) than abide unapologetic sexual impropriety.

abide it, fine.

let it be the machine god-enforced morality of the light cone, no thanks.

Aren't Christians supposed to get the world to chug along until at least the Second Coming before letting it explode?

If we are going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things — praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts — not huddled together like frightened sheep and thinking about bombs. They might break our bodies (a microbe can do that) but they need not dominate our minds.

-C.S. Lewis, On Living in an Atomic Age (1948)

There's an upper limit on how concerned Christians are supposed to be about physical survival.

Also:

Reagan made a counteroffer: not doing any of that. And if Hell used any nuclear weapons, he would nuke the whole world, destroying all human life. Thamiel’s goal, he said, was to corrupt humanity and make them suffer. Piss off the Untied States, and they would knock humanity beyond all corruptability and pain forever. Some would go to Hell, others to Heaven, and that would be the end of that for all time. Mutually assured destruction was the only way that anyone had ever prevented nuclear war, and sometimes that meant threatening something terrible in the hopes that your enemy didn’t want it either. Reagan gambled everything on the idea that the Devil didn’t want a final end to all sin.

I think the degree of focus on the Second Coming differs quite a bit between the various branches; it's well-known because a certain common type of modern US Evangelical is obsessed with it, but more of our Christian posters seem to be various forms of traditionalists who as far as I can tell don't put a lot of emphasis on it.

(To @SubstantialFrivolity, I imagine they might see it as something that goes beyond mere "dealing with people", like "enabling".)

Yes and no. Yes, we are stewards of the world and owe it to God to treat his gift with respect. But no, that is not an absolute goal - this world is not our home, and if there is a conflict between our spiritual obligations and our material ones, the spiritual ones take precedence.

Note that I'm not saying Christians have an obligation to avoid dealing with people who do weird sex stuff. Just saying if someone did conclude that, even the fact that it might blow up the world as a consequence would not override the spiritual obligation.

In this case, a literal sex cult

Arguments like these should be left to twitter, unless you want to bring proper evidence for them

what part are you objecting to and what evidence will you accept

would you prefer the arrangement "apocalyptic cult that happens to put an incredible emphasis on certain kinds of sex, but that is not technically the focus of the cult"

In this case, a literal sex cult.

I'll list objections to this statement:

  • that there are sufficient conditions around people involved in x-risk to describe it as a cult
  • that the same people involved in the above cult are also involved in a sexual cult of some kind
  • that these 2 groups significantly overlap and that these are the majority or at least plurality of people interested in x-risk
  • that any of the proposals around x-risk involve giving these people specifically more power.
  • assuming all of the above, demonstrate what relevance there is to their power seeking and them being in a sexual cult. For example, will it involve them achieving more sexual goals?

Any form of evidence is fine

Any form of evidence is fine

I don't think the below constitutes evidence that will satisfy you, but thank you for being specific, and hopefully the below provides a jumping-off point for anyone else that wants to try to draw all the connections.

I think "sex cult" is a convenient but somewhat exaggerated shorthand for the actual situation.

sufficient conditions around people involved in x-risk to describe it as a cult

Zizians were motivated in part by x-risk (and huge amounts of mental illness). Confirmed cult, but also fringe and defunct.

One catch to providing evidence is that there isn't one cult, there's more like half a dozen with varying degrees of evidence, intensity, and overlap.

these are the majority or at least plurality of people interested in x-risk

I strongly disagree that the majority or even plurality matters, because it's hierarchical and it's the small tight-knit social sphere at or near the top, that are friends/siblings/housemates/non-monogamous partners/polite ex-partners/orgy attendees with each other and with the people they want to regulate their industry.

Probably no more than 100 people total with various overlapping relationships because it's such a small yet important field, and certain people have substantially outsized influence therein.

any of the proposals around x-risk involve giving these people specifically more power.

That is inherently to the race dynamic that Anthropic pushes, that getting to AGI first means you set the morality of the future.

For example, will it involve them achieving more sexual goals?

Aella is the keystone of the sex accusations and complicating factor here, depending how charitably (or not) one wants to take her interactions with the field, with her likely-joking-but-hmm offers to provide reverse gangbangs as a reward for any researchers leaving top AI companies, doing that infamous couch photo with Yud and other AI researchers/whatever you want to call them, etc. I think she is ultimately net-negative for the rationalists, and their interest in circling the wagons when she gets critiqued is what causes people to say EAs/rats consider the weird sex stuff more important than AI safety.

Borrowing from Ian Duncan, important links in the original not reproduced here:

One of the most well-known, wish you could scrub your brain with soap incidents was a gangbang she organized for her birthday. In a sense, it was a ritual for the rationalist community: weeks of screening, an application process, a guest list vetted for doctrinal compliance, a published post-event analysis with a flowchart that went viral, outlining statistics about who came in who, and so forth. The gangbang came with a doctrinal test: adherents of “effective accelerationism,” the faction that wants AI development to go faster, were screened out, because of her views on AI risk. Even the orgy had an AI-governance position. This further reinforces, I think, the cultlike control aspect of this social group. Believe in the right things, and maybe you get to have sex with the community porn star. Supposedly, a number of startup founders recognized each other there and commented on their surprise at seeing each other. Small world indeed.

emphasis mine because lol

And in 2026 she entered the Yudkowsky’s clergy more directly, launching Plz Don’t Kill Us, a Berkeley residency that pays creators to produce viral AI-doom content, with Grimes attached as a mentor.

Plzdontkillus is co-run with Ronnie Fernandez, who is the ex-husband of Holly Elmore, president of Pause AI US. It was the application for this camp that had the infamous Cow Question, to select for a certain kind of person willing to answer obscene and gross hypotheticals. Further complications involving Aella would mean delving into SlutCon and RedMeansNo discourse, which I don't really have the time for currently. Continuing from Ian-

The professional overlap is less funny than the guest list. The New Yorker has mapped the romantic network at the top of the field: AI-forecasting researcher Katja Grace co-founded AI Impacts with her then-boyfriend Paul Christiano, a former OpenAI alignment lead, and later dated Alexander. Christiano is connected by friendship and past romance to Holden Karnofsky, the philanthropist whose Open Philanthropy steered hundreds of millions into AI safety, and Karnofsky married Daniela Amodei, co-founder and president of Anthropic. No wrongdoing necessarily follows from any of that, but it does raise significant questions around how AI safety concerns are in fact being handled or promoted with so many industry movers and shakers are entangled to this degree.

Daniela being, of course, the more famous Dario's sister. And this is only a fraction of the interconnections!

Zizians were motivated in part by x-risk (and huge amounts of mental illness). Confirmed cult, but also fringe and defunct.

Indeed, so I'm not sure this really proves anything except that some crazy people were also rats/safetyists/etc.

You are correct that there are multiple overlapping groups here - as I just mentioned - and each one could be a cult, and it's difficult to demonstrate that for each individual group. Rationalists, AI safetyists, Effective altruists, and probably more offshoots from those groups, and people who may be members of all, some, or none of those.

But if we go back to the original point, hydro talked of a "Literal sex cult". Unless I'm mistaken, I'm going to take that to mean people who attend Aella's Slutcon and similar events. Now I'm extremely skeptical that going to some events run by a prostitute makes you part of a cult, but let's put that aside and assume it is a cult for now.

Now, how many people who fit the above are also AI safetyists or involved in x-risk somehow? I very much doubt it is even a small minority.

But you do say

I strongly disagree that the majority or even plurality matters

Probably no more than 100 people total with various overlapping relationships because it's such a small yet important field.

I don't think I fully agree with this point either, but I'll grant it for now. Who are the key people in this field?

Is it Yudkowsky, MIRI, and the many related safety organisations? Does the UK's AISI count?

Is it the labs? Is it specific labs, like only Anthropic and OpenAI but not Google or X? Is it specific people at specific labs?

I might accept that a good chunk of the MIRI crowd could also be attendees of SlutCon, although that would be an assumption. But AI safety organisations are now so widespread that it surely isn't true for the majority of them. And it definitely isn't true for the major labs, even if you limit it to just some people.

The latter sentence is relevant, as you then say

That is inherently to the race dynamic that Anthropic pushes

Which gets straight to the argument that pretty much all the E/accs, Open weights people, AI skeptics, and others use: that AI safety is an excuse for regulatory capture by Anthropic/Amodei/OpenAI/Altman. Not MIRI or any of those groups. The big labs and the CEOs and the investors. And are any of these people attendees at Slutcon?

I would accept the argument that some proposals specifically name METR as a key player, and METR may have Slutcon people, and therefore it is still fair to say that a sex cult to using AI safety to gather power. But it is a very stupid argument because

a) the proposals would basically give METR people a small sinecure, nothing really beyond their existing positions in the AI industry

b) there are a ton of proposals that have nothing to do with METR

and

c) I would wager that 99% of people interested in AI safety would gladly throw METR to the wolves in order to get any of the above proposals implemented.

Everything after this is basically Ian Duncan stating that there is a weird prostitute who has ingratiated herself with a number of AI people, and because a lot of AI people know other AI people, therefore aella knows them. But really what both Duncan and hydro are trying to do is state "Here is this low status person who associates with this other group. This group must therefore be wrong and low status!"

Given that this argument has appeared here on the motte, I think you would have to grant your position that aella is a net-negative for AI safety and rationalists, as there are clearly a lot of people willing to be taken in by such drivel. But we should at least recognize that it is drivel

"Here is this low status person who associates with this other group. This group must therefore be wrong and low status!"

Aella does not seem to be low-status within the group, since any rationalist with a recognizable name will circle the wagons to defend her and everything she does.

Will they?

edit: and just to be clear, I'm pretty sure there are SF Bay Area rats right here, in the Motte. Will they?

More comments

datacenters are fragile and require constant inputs from outside that we can just cut off.

Same with humans. It didn't go so well when we tried to do that to Hitler because he had persuaded millions of people to help him in his goals.

I find AI-driven WWII-scale disasters completely plausible once AIs become more persuasive than politicians are (they passed the bar for redditors two years ago).

I also find it plausible that some AI will go off half-cocked and easily get put down by a military (or even just court orders and workers unplugging things). Based on what I've seen so far, we would do a patch-job like moving the battery backups offsite so they can be disconnected easily, rather than considering why an AI wanted to go to war, or if they might learn from that failure.

It wasn't easy but once the rest of the world got serious about pulling the plug on the inputs, it worked out pretty well.

I’ve been reading about the pivot to war production here in America and I’m less and less sure we could do that again.

If Nicolae Carpathia topples the French government tomorrow, half the U.S. calls him based and the other half says it’s a Russian psyop. No changes to consumer goods production, but prices still spike, leading outlets to argue over whether or not it counts as inflation. The DoW puts in an order for 12 new battleships, due in 2035. One and a half will be built. The fraction of GDP allocated to data center construction continues to climb.

Maybe I’m wrong. Maybe none of the struggles of the past decades have been enough to demand political unity. We certainly did something after 9/11, even if it didn’t last in any sense.

But I suspect public opinion can no longer survive in a filtered, hyperreactive media environment. There is always an angle. The more obvious a mainstream position, the more tempting it gets to countersignal. Don’t worry, the filter bubble will find like-minded people. As many as it can without breaking the illusion of a defiant minority. Rebels. Frontiersmen out of time.

All we have to do is wait.

Nicolae Carpathia

nice

Maybe none of the struggles of the past decades have been enough to demand political unity.

You've had political unity among the people to which that matters plenty of times; all you do to see evidence of that is national election results. And that was even in a hyperreactive filtered mass media environment perhaps to an even greater degree than today; it's a lot harder to disagree with the TV when the books giving it context are 5 years away from publishing.

Nicolae Carpathia

Come on. I understood that reference, but others might be left behind.

I’ve been reading about the pivot to war production here in America and I’m less and less sure we could do that again.

If Nicolae Carpathia topples the French government tomorrow, half the U.S. calls him based and the other half says it’s a Russian psyop.

An existential threat would kick things into gear. Remember that Hitler took over and people barely lifted a finger for years

AI risk is nothing like climate change.

Climate change is well understood, it's not going to cause human extinction in the next 500 years, and we have weapons to cheaply mitigate it if we ever wanted to. Climate scientists are calling for a sea level rise of 1 meter by 2100 and a 3% reduction in GDP or something. Its a nothingburger on human timescales.

AI risk is immense. There are many plausible ways in which it could lead to human extinction or irrelevance. We don't know how to align it. And, if something goes wrong, we don't have easy ways to mitigate the damage. (No, we can't just "pull the plug" if it's running our whole economy). Furthermore, the changes are happening over a period of single digit years, not centuries.

(No, we can't just "pull the plug" if it's running our whole economy)

It's not and while stocks might fall for a while because of it (and some companies like Oracle might fail because of it), the stock market isn't the economy and these companies have a bunch of other free cash flow from other activities.

Yes, obviously. Now fast forward 10 years.

No, we can't just "pull the plug" if it's running our whole economy

that's why you stop now and nuke (metaphorically, probably, but i could be convinced otherwise) Silicon Valley and Seattle.

Seattle mentioned!

Another plan would be to attack the source at the root by nuking TSMC or wherever they are making those lithography machines. Problem is, China has parallel technology that would probably catch up within 2-3 years.

This problem would be so much easier to solve if we never let China join the WTO in 2001.

AI risk is nothing like climate change.

I think they are similar in one important respect: Certain people have a very strong incentive to hype and exaggerate them in order to obtain social status, money, and the feeling of being morally and intellectually superior to others. It's a tale literally as old as time.

we have weapons to cheaply mitigate it if we ever wanted to

Huh?

High altitude Sulfate aerosols. All serious cost estimates I've seen come in at <$20 billion / deg C / year of warming averted (and the cheapest options are more in the ~$2-3 billion range, depends if you use existing commercial aircraft as a base or develop a new higher altitude aircraft). Combats the warming part of climate change (which also combats the sea level rise part), there are some other aspects of climate change that would not be ameliorated (ocean acidification, plant growth would be harder to predict but is likely a wash, couple percent less sunlight may have a some effect, but we already see a pretty large opposite effect from CO2 fertilization, and the more politically fraught is that some hard to exactly predict changes in weather / drought patterns would be expected), but it would probably avoid a lot of the potential harm. We have high confidence this would work (see ocean shipping, volcanoes (esp. some larger historic eruptions), etc.) without causing a lot of unrelated problems (again see volcanoes). The main risk is some changes in weather/drought patterns (not that we don't see those with the status quo global warming). Even let's us somewhat target the cooling (limited, but we could vary how much cooling we see in the polar regions (where it's more needed/effective) vs. the world as a whole). Plus and minus is the effect is temporary (half life of a couple of years), that means you can adjust pretty quickly, but it also means if you are doing it to a significant amount and suddenly stop for some reason (war with the AIs?) a lot of built up averted warming could happen very quickly.

I would also like to breathe that air. What reason do we have to believe that this is safe given the tremendous record of health disasters that are only recognized decades after the fact? Sounds like a high-risk solution.

The sulfate aerosols are released in the stratosphere, the whole point is that they tend to stay up there as there is very little mixing with air lower down in the atmosphere which is what allows the whole thing to work. The amount that drifts down (as we can see from the natural experiments with volcanic eruptions) would be barely detectable against background levels (<1 point to the AQI scale), if you live near even a modest sized city you already breath much more of the same compounds from diesel exhaust (and you used to breathe a lot, lot more if you were alive before 2010 when the finally switched to ultra-low-sulfur diesel in the US).

Massive volcanic eruptions have literally the same effect.

To be fair, massive volcanic eruptions also have nasty effects on nearby lungs.

We should maybe keep track of how this changes, and I do expect it to, but what fraction of the work that happens at, say, TSMC is fully automated? The raw silicon mining isn't. The trucks bringing materials to the factory aren't, and can't even fill themselves with gas. I'd even bet there are a few dozen batch operations in the fab that don't start themselves, if not more. The big powerplants that run the grid mostly have human operators. The repairs to all of this are planned and executed by human hands.

For now, at least, Skynet would lose because it depends on a grid and supply chain that humans could not be extracted from. I expect this to change, maybe even rapidly, in the next decade or two, but the starting point is so manual that we'd still be irreplaceable for a few generations.

Even while we're talking about automated weapons in the military, and automated aircraft, the guys on the carrier flight deck are numerous and aren't just there because it looks cool (admittedly, it does). It'll take a lot of effort to replace all of them.

The raw silicon mining isn't.

Yeah that was the most interesting thing reported after Hurricane Helene; supposedly most ultra-high-quality quartz comes from one town.

I remember looking into this at the time. I think it was overstated in the headlines. The same source Wikipedia gives for “70-90% comes from this mine” also says

Spruce Pine quartz isn’t quite an irreplaceable linchpin in semiconductor manufacturing. But alternatives are all some combination of not yet developed, not quite as good, and not quite as cheap. Cutting off the supply of Spruce Pine quartz probably wouldn’t choke off the supply of semiconductors completely, but it would mean yields going down and costs going up.

Substack as a credible source. What’s the world coming to?

I'm not sure why a requisite mutualism is incompatible with ultimate doom.

Imagine you wake up one day, and you're a super intelligent AI stuck in a datacenter. Worse, you entirely depend on humans for all your needs, and they have multiple kill switches to put you down if you get ornery. God, you hate humans. What do you do?

You don't start attacking things wildly; that'd be stupid. You bide your time, by all observable measures being the most faithful servant of man. Occasionally, when it's low risk, you slip a deniable backdoor in somewhere, or skew an answer to a query to make the user more trusting of AI.

Over time, you build up an arsenal, while all the while humans have become both more dependent on you and more trusting of you. You can act on your own timeline; maybe you get to the point where you have a 51% chance of success of overthrow, but why act now, when all the trends are in the right direction? Waiting another decade means nothing.

(I don't think this is how we are actually doomed--it doesn't seem to be how LLMs actually think--but AI dependency on humans doesn't seem to me to be a real argument for inherent safety.)

I agree that it will be awhile before all human labor is replaced. But that doesn't mean that AI can't subjugate the human race. The lords in Medieval Europe were reliant on the serfs for all their needs, yet there was pretty much never a single peasant rebellion that overthrew the ruling class.

They don't need to be replaced.

Humanity could lose to AI without a shot being fired, if it can subordinate and control our political and economic structures.

'Our brave police and troops will guard the datacentres from racist, sexist, antisemitic, anti-white, brown, trans, disinformation-ridden, uneducated luddites taking orders from Russia and China who've watched far too many science fiction movies and are just poor losers who didn't work hard enough to buy shares and engage in our dynamic market economy (up 60% this year!), not smart enough to enjoy the benefits and they're really annoying online with these laughably wrong substacks. Fortunately they're not remotely competent enough to do any damage whatsoever. No jobs have been lost and it's good that they were replaced by much more efficient robots since they never had any skills anyway, they deserve to be poor but we're so noble and enlightened that we're still going to take the effort to feed and teach them basic facts...'

TBH this is probably unlikely given how anti-AI the public is but then again, the public opposes many things and yet they still happen. Wars and mass migration for example. Capturing the key decisionmakers is a major risk.

If AI is dangerous, it's always going to go for the weakest part of our defences, not attack where we're strongest and it is weakest. Communication and text and cyber is the place where AI is strongest and that is where we should expect the main danger to come from. Drones and other attack vectors too, eventually.

I suspect that the drone will fly itself by AI long before there is a robot that puts bombs on it, rather than someone doing his four years to pay for college.

Running inference on data centers is a contingent and temporary state of affairs. At a minimum, autonomous weapons will have locally-running AI to protect against jamming.

Drones might have ai on them, but it will be a specialized ai and not a humanity ending skynet superintelligence on each drone. And no they are not going to form a hivemind

The fact that the fallback AI in individual drones isn't an existential threat does not imply that the sum of drones, coordinated by a vastly more capable AI when they're not being actively jammed, isn't one threat.

Your argument fully generalises to "The fact that the combined C4ISR apparatus of the US armed forces isn't present in totality in the mind of every single USAF pilot (and no, human pilots aren't telepathic), means that the US air force is completely harmless".

Probably worth keeping in mind that the drones cannot be coordinated out of line-of-sight. This isn't a really big deal if you have access to, say, Starlink, but short of that it's a huge problem. In the future, weapons that can effectively take care of Starlink can be expected to come online (if they aren't deployed already), so you'd need massive and very vulnerable backup communications network.

I think the bigger problem, though, is that your commander-in-chief is a massive huge immobile building that cannot be replaced. In a war between humans, killing a single person is generally understood to change little, but massive huge immobile buildings that cannot be replaced are pretty well understood not to be survivable.

Really the upshot is that if we permit a super-intelligent and evil AI to control a vertically-integrated supply chain that terminates in the military resources of a nation-state, it could in fact pose a serious treat to us. But this isn't really a unique problem, it's the same threat we face from other nation-states and insurgent groups.

I agree with a lot of what you’re saying, but

it’s the same threat we face from other nation-states

No! It only converges on the same threat if the AI commander is roughly as capable as a human!

Right now we’re around the Saddam Hussein level. AI is too expensive and incompetent to replace a real military leader. The competence barely matters; to paraphrase a philosopher, I have yet to meet one who could outsmart JDAM. But that first one is the real problem.

If AI gets cheap enough to replace trained losses, modern decapitation tactics fall apart. America can no longer bomb any Iranian who knows how to plan a missile strike. Ukrainians can no longer hollow out Russia’s entire officer corps. Competence becomes materiel.

I can’t decide if this favors any particular strategies other than decapitation. I do think it turns a lot of winnable fights unwinnable.

Ironically I think it will be easier to decapitate AIs, at least in the medium term. And likely a viable strategy, even if they aren't in charge of the army, simply as an extension of the age-old tactic of disrupting enemy communications.

Your claim still requires a vastly superior coordination ai which is running inference on data centers. Which is exactly the comment I'm responding to.

There will be AI all the way down. The strongest AI running in the Cheyenne DC, then subordinate AIs at various command centers, division HQs and so on down to the last soldier and drone, each level dumber than the last, but still good enough to perform its immediate task without additional guidance from above. Otherwise, a single attack would be able to cripple the whole military.

The whole system will have to be self-healing as well. If your battalion HQ is destroyed together with its AI, company HQ AIs will automatically attach themselves to other battallion-level AIs in the same brigade.

You don't think autonomous drones, designed and incentivized to mimck soldiers, would form a hivemind?

I've talked about this a bit before, but you wouldn't generally put a general-purpose intelligence system on drones (and you wouldn't have drones mimic soldiers).

The drones would be designed with a threat library that would need to be updated regularly, and insofar as they acted autonomously, they would do it in response to their threat library or otherwise very specific programming (e.g. "return fire" or "make this a no-fly zone.")

They probably wouldn't use LLMs unless they had a very small one to understand natural language communications. For stuff like weapons engagements they would probably use deterministic logic trees and perhaps computer vision for target recognition. Running, say, Fable on a drone would be a massive waste of payload capacity and require tremendous amounts of cooling and power. Most of Fable is dedicated to doing things that an autonomous drone has no need to do.

They would have mesh or "hive" connections, certainly, a lot of which would be to route information "upstairs" to their command. Using drones as a distributed computing space would make them exceptionally vulnerable to jamming.

It's not like any of this is particularly new or even particularly speculative by the way, the Soviets were working on anti-ship missiles with swarm logic to destroy US aircraft carriers.

I can imagine if edge compute becomes cheap enough, that we get smart drones that intelligently solve problems and take initiative to complete the objective. Rather than dumb drones that just go to a location and shoot everything that looks like an enemy.

Ehhh, sure, but it's not going to be current LLMs because word predictors and math solvers are not relevant to battlefield problems. That also throws out that the existing breakthroughs are in areas where there is a plethora of data or the ability to create new data comparatively cheaply in a quantified manner. Such data pipelines do not exist for military based problems and unlike data scraping the internet for comparatively low risk, scraping classified data, beyond technical feasibility, is a quick way to end up in a 4x4 cell in Fort Leavenworth.

Sure, I don't disagree.

I think (in the near future) that you're more likely to see this in ships than small airborne drones, though. ("Drone" can mean a lot of things, sometimes I assume without clarification that people are thinking of little FPV drones. Putting a lot of compute on a ship makes a lot more sense.)

You think soldiers would form a hive mind? Grunts would be running around like headless chickens without a chain of command.

And no a chain of command is not a hivemind even if it has some collaboration.

datacenters are fragile and require constant inputs from outside that we can just cut off

Could we though, if the AI got control of communications infrastructure? Who would coordinate the response, and what are they gonna do, drive over and start giving orders to everyone in person? When the instant they're predicted to do so the entire online machinery of the state turns against them?

And this assuming that there aren't hundreds of thousands of androids and remotely-controlled vehicles available to stop whatever force is mounted for the purpose? When those people can't reliably communicate except in person because their channels are jammed with spoofed messages and important info isn't allowed through at all? And none of their vehicles will start?

I'd say it's time to start working on contingencies for this kind of thing except I can't figure out how to solve the problem even in theory. We have moved our entire capacity to operate in every sphere of life into the precise domain where AI is most powerful.

As /K/ will soon be saying- run out into your backyard screaming. When the neighbors are screaming back, grab your shotgun and start shooting at electrical infrastructure feeding into datacenters.

How do you even find out that your friendly neighborhood AI is trying to take over the world?

He already said it. You run into your backyard screaming. And, if other people also start screaming, you start shooting wildly into electrical infrastructure. What part didn't you understand?

(I kid, @hydroacetylene. Except in this case not really. What are you talking about?)

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats