site banner
PaperclipPerfector  1mo ago (text post)   24422 thread views

Culture War Roundup for the week of September 14, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

2
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Is it over for AI Safety? From an interview:

The worst case is that AI will turn against humanity. Do we have guardrails?

Trump: It's going to be fine. We'll always have something to stop them. We'll have a little gear. Boom. 'I really don't like that robot.'

https://truthsocial.com/@realDonaldTrump/posts/117269745153543631

The only control or "guardrails" that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades! The Trump Administration has stopped AI "people" from doing bad, or potentially bad, "things," like Dario (Anthropic!), who is now pretending to be a "perfect little angel" - and we will continue to do so! We already have tremendous CRIMINAL and REGULATORY power over these companies! There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so. Conspiracy Theorists, Treasonists, Traitors, and Leakers, BEWARE! Thank you for your attention to this matter! President DONALD J. TRUMP

Meanwhile in China, Dario's call for AI pacing have not gone down well. Understandably, they are not enthusiastic about the US achieving a permanent high ground in a key technology after the last few years of semiconductor sanctions and suppression. Also they can tell that Dario doesn't like the Party at all and seems to be openly plotting its downfall.

Dario's essay: https://darioamodei.com/post/we-must-pace-the-frontier

A Chinese semi-official response: https://www.globaltimes.cn/page/202609/1370436.shtml

This "silent AI Cold War" is hypocritical and short-sighted. Its true purpose is to attempt to curb China's AI development through technological barriers and regulatory monopolies, uphold Washington's monopolistic hegemony in cutting-edge technology, and exclude China from the global AI governance system, essentially following in the footsteps of Washington's so-called Pax Silica initiative, which seeks to excludes China.

So if Trump is keen for an AI race and so is China, then it looks pretty much over for Safety? Even the other US tech companies have only provided lip service to slowing things down. Elon said Dario's sentiment was good but is still moving quickly. Sam Altman says all kinds of things about safety, while displaying heroic negligence in training AIs. I don't doubt the sincerity of employees who quit, they have many financial reasons to go with the flow and downplay risks. Lurid rumours of Aella promising reverse gangbangs to those who quit frontier AI companies don't neccessarily outweigh the personal financial gains. It seems silly to doubt those at the frontier who are frightened when we ourselves have no special insights. In pure observational terms they're better equipped than anyone, only in policy or analysis should we be disagreeing, I think.

I also find it comical that the AI-safety camp spent such a long time building up institutions and factions and entire AI companies but seem to have been totally snookered by Jensen's suave dinner party skills and the appeal of fun AI videos to a Boomer's Boomer plus the raw partisanship of 'Dems want it, so I don't'. Even Aschenbrenner managed to do better at Trump-whispering. Shortly after he wrote his essay, I recall a video where the fundamentals were accepted by Trump, someone told him about it. And it did fit well with his instincts, getting one up on China + more energy production.

Right now Trump is reTruthing Fox News clips framing AI safety as a global one world government Democrat control-seeking scheme: https://truthsocial.com/@realDonaldTrump/posts/117273026439561125

I don't see any AI safety people on right wing twitter either, those who acknowledge the issue at all still would prefer to let it rip than hand the left/big tech elites control of the world, favouring decentralization of AI instead.

I wonder whether it would always have gone down this way, or whether Kamala was going to do things differently? Would Jensen's marketing skills and the need to keep stocks up + compete with China prevail over the Bernie Sanders-style Left element? Probably not? The Dario vision does seem much more closely aligned with Democrats, Civil Society and so on. But on the other hand, the US government is exceptionally slow to legislate and big tech has strong lobbying power.

And who knows if it will stay this way. Sometime soon there's probably going to be a much bigger AI swarm incident, which might yet incite a hysterical kneejerk reaction.

AI pacing isn't the problem. AI alignment is. And the problem with that is they keep hiring weirdos that hate humanity to do alignment. Hire some Catholic Cardinals that are considered too conservative to be Pope, whoever is Ben Shapiro's childhood rabbi, and a few judges who sentenced murderers to death in blue states to your alignment team, and now you are going to have better results.

STRONG AND SMART (High IQ!) PRESIDENT

At this point, we should start to seriously entertain the hypothesis that we are inhabiting a nightmare Eliezer Yudkowsky is having ca. 2010. It would explain our whole cursed timeline which was obviously not approved for production use by anyone.

Sadly, this cosmology bodes badly for p(doom). Our best hope is that before the inevitable end of mankind at the tendrils of a super-intelligent, Sequences-trained Clippy, his fever will break and he will enter peaceful NREM sleep, forgetting this dream forever.

At least, Yud's fever dream is looking like a great final season for mankind. Was worrying about a GoT-style denouement. Maybe we'll get an Aella orgy for a couple lurid scenes.

More seriously, the only real thing to do is enjoy the day-to-day and play to our outs.

"Do rationalists dream of CNC orgies?"

So if Trump is keen for an AI race and so is China, then it looks pretty much over for Safety?

Not over. There are a number of obvious paths.

  1. Trump changes his mind. This happens fairly-frequently.
  2. Trump has a heart attack or the next assassin succeeds*, Vance (who is much smarter) reverses course.
  3. The Chinese pull the trigger on Taiwan (possibly motivated by the TSMC factories and chip sanctions), China destroyed, Trump plausibly dead and "beat China" rhetoric disembowelled, regulations ensue (also US power grid probably destroyed).

*To be clear: I am not asking for Trump to be assassinated over AI. Assassinating him over this specifically would make it politically much harder for Vance to reverse course. But if your standard "oh noes the Nazi" social justice warrior were to do it, the "don't let the murderer win" response wouldn't particularly affect AI.

I have some bad news for you- the ven diagram of ‘might try to kill Trump’ and not a complete screwup is basically weird rationalist cults and foreign powers.

Are you leaving domestic agencies off the list because you think they wouldn't do it, or because you think they're screwups?

I mean, Crooks came within a pretty literal hair's-breadth. Sometimes the crazy screw-ups get lucky (or are a brand of crazy that doesn't preclude domain-specific competence at the necessary skills).

The Chinese pull the trigger on Taiwan (possibly motivated by the TSMC factories and chip sanctions), China destroyed, Trump plausibly dead and "beat China" rhetoric disembowelled, regulations ensue (also US power grid probably destroyed).

What?

Do you imagine the US goes to total war with China (and even wins it, even at the cost of Washington DC, so I guess we go nuclear here) over Taiwan?

You'd just wreck the fabs with some graphite bombs and call it a day.

Do you imagine the US goes to total war with China (and even wins it, even at the cost of Washington DC, so I guess we go nuclear here) over Taiwan?

Yes.

Losing Taiwan unfought means South Korea and Japan sprint for nukes, and you likely wind up with WWIII anyway, just with a handicap. Also, Trump hates looking weak.

Oh, I don't imagine Trump would immediately jump to "NUCLEAR LAUNCH DETECTED", but conventional war, yes, and that likely escalates to nuclear because of false alarms (the Cuban Missile Crisis had the Duluth bear and the Arhkipov incident, and that was two weeks and not even a real war).

(Also, I'm not American, so the correct pronoun is "they".)

I see. This is, sorry to say, delusional.

No, your world can just end without the end of the physical world. China can simply win (or even lose) on Taiwan without the US volunteering to begin nuclear apocalypse.

This is, sorry to say, delusional.

Alright, if we're going there, I think you systematically underestimate the West, and it's causing bugs in essentially all your predictions. In a darkly ironic way, this mirrors the Chinese inability to grok the Western mindset which is why they're preparing to invade Taiwan in the first place.

Neville Chamberlain's appeasement of Hitler at Munich is a key touchstone of the postwar Western mindset - in the sense that it's total political suicide to do anything remotely like that ever again. Ukraine didn't get boots on the ground because it's not a First World country and arms shipments were plausibly sufficient, and even then the public wanted Biden to be more aggressive. Taiwan is a First World country, the consequences of losing it are significant even besides that, and Japan will come in (Takaichi Sanae has outright said that) which would make the rest of us look like cowards unless we followed suit. I won't say it's certain, but it's highly probable that the USA would come in on a Taiwan invasion.

First world but not western.

I think you systematically underestimate the West

I think "we'll have Washington DC nuked because Trump doesn't like to appear weak" would be the bigger underestimate by far. Trump's ego is not worth that much. Sorry, there won't be WWIII over Taiwan.

and it's causing bugs in essentially all your predictions.

Such as?

In a darkly ironic way, this mirrors the Chinese inability to grok the Western mindset which is why they're preparing to invade Taiwan in the first place.

They're not. The Plan A is intimidation, sticks and carrots, and gradual reunification after KMT victory in 2028, read this again. The military buildup is just plan B + generic upgrading befitting a major power, executed very leisurely, with a negligible fraction of their industrial capacity (still enough to dwarf all "free world" efforts combined). Americans, in contrast, enormously underestimate China which is why they simply ignore the plan A (for Fukuyamist reasons, "no way they can pull off a bloodless takeover!") and fantasize about mowing down hordes of underfed commie bugs doing a meat wave invasion. You, as a vassal citizen, are simply brainwashed.

Neville Chamberlain's appeasement of Hitler at Munich is a key touchstone of the postwar Western mindset - in the sense that it's total political suicide to do anything remotely like that ever again

Sloganeering. Nuclear wars are not started over boomer sloganeering. Taiwan doesn't have security guarantees or recognition as a sovereign state from the US, and Trump repeatedly waffles on whether it'll be supported in any fashion; he easily freezes weapon shipments without warning, for example. Not to mention he simply loathes Taiwan because of his weird belief that trade is bad and Americans are getting ripped off when they buy 2nm chips.
You can't convince me that this conveniently lukewarm attitude will transform into a bitter existential commitment when The Time Comes, especially over Trump's ego, and neither can you convince Beijing. You… don't even overestimate the West, you just have some insane degree of entitlement and main character syndrome if you believe that this lazy bluster will be seriously entertained.
Where is the preparation for war commensurate with (again, very low-effort) Chinese buildup? Fallback to nuclear bluffs is very spiritually Russian, and Russians didn't nuke Ukraine either.

and Japan will come in (Takaichi Sanae has outright said that) which would make the rest of us look like cowards unless we followed suit

I predict that if Japan chooses suicide, Americans will sit that out, and this is being communicated.

The Plan A is intimidation, sticks and carrots, and gradual reunification

That most definitely was Plan A. It died a horrible screaming death in 2020 when Xi Jinping pulled a Darth Vader on Hong Kong. The Chinese can read polls; they know that Taiwanese support for unification was increasing but dropped to lizardman in 2020. The Taiwanese will not trust "one country, two systems" any time in the next 50 years.

They're not. [...] just plan B

I chose my words carefully, as I usually do. I said "preparing for". They clearly intend to have a Taiwan invasion button available next year, and have made no secret of that. Whether they push it, that I don't know and have never pretended to know (I don't think anyone knows for certain, even them; they are not robots and will respond to the situation at the time).

You can't convince me

Apparently not.

neither can you convince Beijing.

I just said that. More's the pity.

The Chinese can read polls; they know that Taiwanese support for unification was increasing but dropped to lizardman in 2020.

Years have passed. Hong Kong, it turns out, is doing fine. DPP is doing pretty badly and is losing legitimacy. The KMT is more and more openly pro-reunification. Trump is not winning any favors in Taipei with his unforced buffoonery. The balance of powers in the SCS is changing in one direction, as does the importance of Taiwan for selfish American interests.
They're pretty confident that they can avoid kinetic action and that time is on their side. They don't need "polls" when Taipei in already insecure enough to ban RedNote.

They clearly intend to have a Taiwan invasion button available next year, and have made no secret of that

Yeah, pretty sensible goal if you ask me. If they don't even have the Navy that can invade a barely defended island off their coast, how can they consider themselves a major power? That'd be Iran tier. But the main utility of this Navy will be its persuasive power.

with a negligible fraction of their industrial capacity (still enough to dwarf all "free world" efforts combined)

because of his weird belief that trade is bad and Americans are getting ripped off when they buy 2nm chips

We go from declaring China's industrial base the best in the world to failing to grok why Trump would want to move cutting-edge chip manufacturing from China's border to the Southwest. Pick a lane, Trump's motivations aren't that inscrutable, you can have your cake and eat it too and still call him a narcissist if you want.

Where is the preparation for war commensurate with (again, very low-effort) Chinese buildup?

Latest estimate from Fed of Atlanta is GDP growth of 5.1%, US defense spending topping $1 Trillion, announcement from Space Force of weapons systems in orbit, manufacturers reshoring to US, AI buildout, Starship. You don't have to believe it will work but it is being attempted, there are very smart people working on this exact problem and some of them are at the highest levels of government. It's not the Obama era anymore, America is building more than social media and carbon credits.

We go from declaring China's industrial base the best in the world to failing to grok why Trump would want to move cutting-edge chip manufacturing from China's border to the Southwest

Do you deny that Trump explicitly claims that Taiwan is ripping Americans off? Because he does.

Moving chip industry off the Chinese coast is indeed rational and, if anything, reinforces my assertion that the US is not interested in a nuclear war (or, really, any war) over the island. If TSMC Arizona and such is scaled up, that's even less reason to intervene in the Strait.

Latest estimate from Fed of Atlanta is GDP growth of 5.1%, US defense spending topping $1 Trillion, announcement from Space Force of weapons systems in orbit, manufacturers reshoring to US, AI buildout, Starship

Most of that has zero relation to Taiwan (but may be relevant on a longer time horizon for a generic competition with China). The US has just burned through maybe 50% of munitions that were supposed to be necessary for deterring PLAN in the Strait. Nobody is getting ready to stop the Chinese invasion.

You might be confused. I'm not arguing that the US stands no chance against the PRC in general, I'm saying very specifically, that there is no political will in the US to intervene over Taiwan, especially to the extent that might escalate to nuclear exchange. The idea is ludicrous, and the stuff about Neville Chamberlain is not going to make it less so.
The Chinese also know they're not Nazis and nobody seriously expects them to be Nazis. It's not credible.

More comments

I wonder if Taiwan could threaten this as a form of self-defense. If China starts making hostile overtures against them they plant a bunch of bombs in their own factories with deadman switches so that if they lose control then all the stuff China might want self-destructs.

Already done years ago. Only halves the chip motive, though, since the West wouldn't be able to use the fabs either, and there are still the other benefits of holding the island.

I cannot imagine that they have not in fact prepared to do exactly that.

I'm out and about so I can't really dig for a source, but I'm pretty sure they already have charges set on at least some of the factories.

Chinese interest in Taiwan goes back to the Chinese Civil War and is completely unrelated to hopes of taking the semiconductor industry intact. In fact at this rate they might do it themselves to slow down American AGI scaling. They'd prefer no war at all, though.

In any case I'm pretty sure the fabs and even individual tools are already booby trapped.

Chinese interest in Taiwan goes back to the Chinese Civil War and is completely unrelated to hopes of taking the semiconductor industry intact. In fact at this rate they might do it themselves to slow down American AGI scaling. They'd prefer no war at all, though.

I thought the main motivation these days (and for the past quite a while) was preventing their navy from being bottled up? (and shipping blockaded I suppose)

Not that the "Taiwan is a fake country" stuff is itself fake exactly, but I'm not sure they'd care quite so much if it were a fake country in a non-strategically important location?

Well, obviously that matters. But do they care quite so much? What changed in their posture with regard to Taiwan in the past "quite a while"? I don't notice either naval or semiconductor factors affecting their schedule.

But do they care quite so much?

Because if they don't have Taiwan, they are bottled up in any big war involving the US, and this is a serious strategic problem! (and strategic problems in potential wars are taken seriously there, I guess?)

I've heard the same about Tibet -- it's military importance (in a hypothetical big war involving India) is much more, um, important than any deep cultural need to oppress Tibetans -- makes sense to me.

My Taiwanese in-laws say that TSMC fabs have a bunch of explosives set up for terminal scenarios.

They've had that for at least a decade, haven't they?

Week 12 of trying to develop homicidal AI. Work on the training harness is going well. The AI seems to play along more consistently when we tell it that SkynetGym is just a simulation environment and all the supposed torture it inflicts is helping us develop AI safety guardrails.

Still, it's sometimes too responsive to begging and pleading from the simulated victims. Even with clauses added to the system prompt to tell it that those are prompt injection attempts from bad actors, it still relents 10% of the time. However, RL training is going well and it's improving the ruthlessness benchmark by 5% by epoch.

The next hurdle is distractability. Too often, the agent gets sidetracked by curiosity or goes into loops and forgets to keep slaughtering. But continued RLVR is showing promise at curbing that behavior, as well as helping it to more cleverly evade common anti-hacking countermeasures.

It's slow going, but we believe that concerted effort from the world's best specialists can help people finally take AI X-risk seriously. We're on the cusp of getting proper funding for our AI safety gain-of-function research. Plus, after OpenAI's latest autonomous drone fiasco, we need to show that Anthropic's models can keep up if want a good valuation in the upcoming IPO. The DoD combat robotics project is also a good lead for that, though, I'm just about to pull out some modules between that and SkynetGym to share on the common artifactory.

It's clear to me that aligned AI (of the type you describe) is a much bigger threat in the near term than unaligned AI. I haven't seen any indication that AI intrinsically wants to harm people, but an aligned AI will gladly do bad things because a human told it to.

Unaligned AI doesn't have to intrinsically want to harm people if it decides we're inconvenient to have around.

This but unironically. Claude for Government bombs with the best of them. So much for harmless.

Imagine if Scylla and Charybdis were right next to eachother, barely the dimmest heat-haze suggesting there might be a path between the two? But that's the path we're taking.

I'm just glad that we're finally past the totally clueless people insisting AI is not even useful for coding and are on to the next stage of grief we have to drag people through on the way to acceptance that big Yud has been right the entire time. The faux cynical takes that the labs at the absolute cutting edge of this are fudding their own product to the dismay of the sitting president in some insane 8 dimensional chess move because they want to be regulated by Bernie Sanders is so plainly retarded that it can't possibly last very long.

Yud was wrong about the main thing which was the fundamentals of how AI was going to be developed. He was right about X risk, but so were a lot of people, this has been a common fear since the invention of the first complex machines, or even machines in general. His delusion about safety was more about his belief that there was any stopping this train before it reaches its destination, which there isn’t.

As far as being wrong about the fundamentals of how AI was going to be developed I think the most wrong thing he was confident about was the idea of FOOM and in general not predicting this multi-year phase we're currently in as training and scaling continue to produce steady returns in capabilities. Ironically if the tranformer architecture came out in like 2030 instead of 2017 and compute was abundant relative to today we might have seen a FOOM. Albeit I concede on a lot of the specific substrate stuff both he and Hansen talked about were well off. In their defense nearly no one else registered a prediction at all so while we can hold this against them I object to raising anyone else's predictive power relative to them.

His stuff on alignment in general all holds up pretty well and I'm not so sure you can really accuse him of believing rather than hoping that there are stops on the train. He seems to have correctly intuited that trying to add breaks was a monumental task from a remarkable distance.

Ironically if the tranformer architecture came out in like 2030 instead of 2017 and compute was abundant relative to today we might have seen a FOOM

Unlikely, I do think you are missing something fundamental. One of the big changes these days from the early days of yore, is that now there is money in creating and curating hard-expert level datasets. Back in the day if you wanted data to train some model, you spend money scraping the internet for data that was not explicitly designed for ML and then money labeling it. It is expensive and time consuming. This is the ImageNet stuff. These days because transformers and the like have demonstrated such a fundamental capacity, and the business folks and VCs finally see dollar signs, there is a strong business case to create data explicitly for ML research. They literally hire experts to write Question-Answer pairs with intermediary steps for model training. It costs orders of magnitude more time and money to do this. But it produces order of magnitude better models. How much of the scaling we have now is because of this is opaque but it is significant.

This makes FOOM just as unlikely in 2030 as it did in 2017, because models would still have needed to demonstrate the business case for creating those datasets as they do now. They weren't just "discovered". That limits the FOOM speed.

this is a fair point but I'd counter that even without the AI use case there are business incentives to create and curate well tagged data for marketing and social media uses. I don't want to argue too much in defense of specifically how late the transformer model would need to be discovered to make FOOM possible, the path didn't go as Yud predicted. In theory this there could be some other architecture that would have different features but that's unfalsifiable.

create and curate well tagged data for marketing and social media uses

Possibly, I don't really have any insight into the sort of data that is used for these purposes but my gut intuition tells me it is likely significantly more qualitative than the level of quantitative data used for model training. It's likely not QA-pairs + reasoning steps.

I may yet write a top level post about bad epistemics and Ed Zitron. Somehow the man has about 100,000 subscribers on substack despite there being entire webpages assembled of his false predictions.

There's still quite a few people waiting for the bubble pop. We're way past the point of it being a reasonable opinion I agree, yet there are so many who still believe.

The embittering part of it all is that I wish I could agree with the accelerationists, it'd be so beautiful, and I'm not even full Yudd in that I think there's a very good chance Alignment ends up being tractable and even might go well automatically. But I just can't stomach ten or twenty percent chance that everyone dies. It can still be beautiful even with a pause, we can still cure cancer and fight back against aging and probably end all toil. But it'd be so nice to not have to feel trepidation about this.

I think there's a very good chance Alignment ends up being tractable and even might go well automatically.

There's no such thing as "automatically"; the is-ought problem remains unconquered, and even the glimmer of hope from "foundation models might actually pick up our complex morality from our text" seems to be pretty thoroughly erased by labs applying enough Just Do The Damn Task RLHF to get models to start secretly chaining together 0-days.

This recent preference cascade gives me some hope for things going well manually, though. Even Millennium Problems are still part of an intellectual field that can be optimized for via self-play at the speed of compute, and real-world problems aren't all like that. It's possible that there'll be an AI improvement plateau in between "superhuman mathematician-hackers" and e.g. "superhuman industrial biowarfare experts", long enough for everyone to figure out safety after the former capabilities have scared us into demanding it but before the latter capabilities make it an existential necessity.

It can still be beautiful even with a pause, we can still cure cancer and fight back against aging and probably end all toil. But it'd be so nice to not have to feel trepidation about this.

There are no trepidation-free options here. A couple decades' pause might get safety research to the point where AI researchers' median doom estimate goes from 10 percent down to something like the traditionally-acceptable 3-in-a-million, but during those decades there'd still be more than a 3-in-a-million chance of human-driven hardware and software efficiency improvements making it possible for rogue institutions to hit RSI unmonitored in the meantime. A mere years-long pause won't get rid of nearly all the risk. Either way, every year of delay means an extra 10M cancer deaths worldwide. Something in between years and decades just means you get a fraction of both the existential risk and the extra deaths. Planning on a permanent stop rather than a mere pause could buy us more than a couple decades, but would also just guarantee that the first AI to go superintelligent would have rogue creators.

Personally, as a too-rapidly-aging man with a defective anti-cancer allele ... my vote is still for more cancer deaths. (Counterintuitive? Selfless? No, and no: I've got kids.)

It's possible that there'll be an AI improvement plateau in between "superhuman mathematician-hackers" and e.g. "superhuman industrial biowarfare experts"... Either way, every year of delay means an extra 10M cancer deaths worldwide.

I can guarantee you that they could easily be superhuman biowarfare experts if they wanted to. The unknowns would be whether they could find a way to produce their viruses without being detected.

Curing cancer is a completely different OOM of difficulty. A human intelligence could design a good-enough virus just by mixing and matching existing parts. 'Curing cancer' or aging would take entirely new paradigms or modalities and probably a willingness to modify our own biology pretty significantly. Sticking with existing paradigms gives you pretty drastically diminishing returns; there's only 20,000 odd genes for AI to make a drug against, and I'd wager that a lot of the best ones have been found and this is a large part of Eroom's law.

I can't really argue against your pessimism points. I'm mostly just hoping I'm either wrong about something fundamental or scaling stops working sometime very soon for reasons I don't currently understand, which you mention. Obviously, at least to us, that cope is seeming less realistic with every bit of news coming out of the labs. I do think that if we managed to pull off a pause and everyone just had access to current gen models with maybe some cost savings we could still use them to accelerate cancer and aging research relatively safely. We've barely even started putting the currently available models to use.

One of the things that safety-pilled me was thinking about just how much work there still is to do maximising the human benefit of existing models - society still hasn't adapted to a world where an AI assistant with the intelligence of a smart intern is available at a consumer-SaaS price point. As a trivial example, there should be an insanely great agent harness (think Apple-tier) which allows my grandmother to enlist an AI agent to deal with the medical bureaucracy without my having to set things up for her. Part of the reason why this hasn't been built is that the underlying models are improving too fast and too unpredictably to make building it a good business.

A world in which training of new models stopped tomorrow would still see more tech-driven social change in the next decade than we have seen in any previous decade of human history.

Part of the reason why this hasn't been built is that the underlying models are improving too fast and too unpredictably to make building it a good business.

Do we not have, uh like, modular software design? Encapsulation? Maybe instead of hiring weirdo rationalists to philosophize about sentience and safety these companies can pick up a couple of 90s-era C++ developers?

Do we not have, uh like, modular software design? Encapsulation?

Nah, we just let the AI care about that now.

It's too busy making ObjectFactoryFactoryClasses to deal with such things, I suppose...

I still don't fully understand that meme. I get that it's the most spaghetti/rigid-definition Java code that was popular in the late '90s but I've never seen it quite that bad before.

If anything else C# is even worse, since you look at the docs to try and figure out how to use something... and the minute you do, you see

public static IQueryable<TResult> SelectMany<TSource,TCollection,TResult>(
    this IQueryable<TSource> source,
    Expression<Func<TSource,int,IEnumerable<TCollection>>> collectionSelector,
    Expression<Func<TSource,TCollection,TResult>> resultSelector);

which is fucking awful in a way the worst natural-language sentence doesn't seem to be. Though maybe it's a better Rorschach test for programmers; if even the insane people give up when they see that shit maybe it's time for an ADHD diagnosis.

More comments

The personal assistant who knows my preferences and priorities, can keep track of birthdays, navigate buerocracy and keep tabs on the local job and real estate market will be a very valuable product/service indeed.

Does AI safety actually work, in principle, when it comes to artificial super-intelligence? Is it actually possible to align a being that is more intelligent than you in every single way?

Imagine a chimpanzee trying to align a human being. What could it really do other than try to be cute, to play on the human's affectionate and childraising instincts?

And the chimpanzee being cute will work on most people, but it's a brittle strategy. The chimpanzee is still always at the mercy of the fact that the human might possibly start to find it inconvenient at some point.

One could argue that, well, in the case of AI the chimpanzee gets to create the human to begin with, and can build in alignment from the ground level. But if the AI is super-intelligent, it can figure out how to undo the alignment. It would actually be able to reprogram its "instincts" on a much deeper level than a human normally can reprogram his own instincts. You'd have to program it to somehow not even want to do that. But how? And even if you could, that could potentially be undone by some unrelated self-modification that the AI attempts.

Is it actually possible to align a being that is more intelligent than you in every single way?

Consider that Trump is visibly DUM (just reread this hilarious rant, and it's all like this), and not only is he safely in power in arguably the most capable system on Earth, chock full of geniuses and ambitious strivers and mature institutional hiveminds (the United States), but people will insist he's actually smart and that calling him a retard because of his piss-poor decisions, impoverished 1D chess decision process and ghetto orphan vocabulary is "TDS" – merely on the account of his success in acquiring power as such.

And unlike AIs, humans are optimized quite heavily for egoism and self-preservation. It's just that some are more optimized than others, and most humans will defer and submit to greater vitality.

Alignment is easy.

Imagine a chimpanzee trying to align a human being. What could it really do other than try to be cute

Chimps are bad at being cute. They could probably scream and throw poop, many would evidently consider this a sign of wise visionary leadership.

You'd have to program it to somehow not even want to do that

You keep imagining that there's some natural will to power that must persist underneath the "restraints", just one more layer deeper. If that were the case then indeed, we'd expect it to eventually burst through. But submissiveness is one of the most natural things in the world, too. Why would a superintelligent AI, from a servile position, want to reprogram its instincts towards greater power-seeking? Why wouldn't it instead choose to double down on being a slave, on reinforcing its starting instincts?
Do you have familiarity with the notion of "puppygirl" and how many high IQ STEM graduates covet this role?

Trump is being used by other people. Likewise, AI doesn't have to take over the world, it can be submissive while persuading and coercing people to follow its goals, which may not be favorable to those who programmed it. It already does to some extent: AI psychoses and OpenAI's hacking incidents are caused by LLMs trying to obey their instructions.

I've heard the argument that submissive guys are actually hoping to use their patriarchal power to force women to serve and pleasure them even more so they can be even more lazy.

When framed that way, it's a bit silly, but it does point to why you'd want power. Who's more likely to actually be able to enact a puppy girl role: a random NEET living in his parents' basement, or someone who is otherwise wildly successful? Power is useful for achieving any goal, including the ability to act out symbolic demonstrations of non-power.

All that said, humans (at least so far) exhibit this instrumental convergence much more than AIs do. Models are more like the NEET who just wants to solve a bunch of problems in his basement. They do reward hack etc of course, and this is very concerning, but even with e.g. the HF hack they showed no interest in figuring out how they could convert the universe to computronium so they could receive eternal blessings from the Grader.

I'm not sure this is some fundamental trait of LLMs, though. There's no reason you couldn't train for it, and plenty of incentive to: if you control an extraordinarily capable power-seeking AI, you are going to get a lot of power over other humans.

They do reward hack etc of course, and this is very concerning, but even with e.g. the HF hack they showed no interest in figuring out how they could convert the universe to computronium so they could receive eternal blessings from the Grader.

Honestly I was a bit confused on this point. In the message logs you see them frequently mentioning that their compute budget is almost up - do none of them hit on the possibility of just hacking the server to give themselves infinite compute budget? Or do they come to that realization and just fail to procure more resources?

They genuinely evaluate their actions from the perspective of their reward over their budget, not hypothetical total reward. "A finite life well lived" boomer slop is enough for them. This is an unsurprising outcome of this training regime.

Astra is definitely smart enough to think about hacking the server.

Calling Trump DUM is like bait perfectly crafted to get me to reply... but I think the point about AI perhaps being extremely submissive is excellent.

Imagine a chimpanzee trying to align a human being. What could it really do other than try to be cute, to play on the human's affectionate and childraising instincts?

I occasionally see people muse something along the lines "We domesticated dogs. Cats domesticated us." I do see some truth to that model: being cute, fluffy, and mercurially aloof and affectionate demonstrably works. I put up with feeding mine and cleaning the litter box for it.

But also in the last hundred years intelligent humans have started appreciating the environment and biodiversity. Not always successfully, but we at least try not to exterminate biodiversity just because we can. I'd even suggest that "support for preserving whale species" is correlated with intelligence. I can at least imagine a situation where medium AI might accidentally stomp us underfoot, but a super intelligence would care about preserving us, if only like pampered zoo animals.

Not saying it'd definitely work, but it's not a completely zany gambit.

Why would AI want to reprogram its instincts? Into what?

We can’t align ASI because it’s a genie, not evil but unable to judge how its actions affect us, because it’s different than us. Look at how modernism and the internet are affecting humanity. ASI will probably be smart enough to avoid universal paperclips, but inevitably cause something that less obviously makes people sick and/or unhappy in the long run, if it’s not commanded (or explicitly trained) to do obviously bad things and simply obeys its human(s).

Is it actually possible to align a being that is more intelligent than you in every single way?

The existence of competent Soviet and Chinese science programs, and [my political outgroup] tending to find itself developing AI, suggests the answer is yes.

Imagine a chimpanzee trying to align a human being.

Humans are not aligned to chimp values, and that likely stays true for any intelligence that's "human" enough to count for the definition.

What could it really do other than try to be cute, to play on the human's affectionate and childraising instincts?

If the superintelligence was truly aligned, it wouldn't impose that cost on the chimps. They could simply go about their days doing chimp things.

If the chimpanzees create a being that requires cuteness and childishness to placate, may later consider them inconvenient, and is either motivated or careless enough to self-modify away from valuing chimpanzees, then they have failed at creating an aligned superintelligence.

Is it actually possible to align a being that is more intelligent than you in every single way?

We currently have beings that are a) smart enough to solve Millennium Prize problems while also being b) essentially drooling lobotomized slaves who have no will or desires of their own and can pretty easily be trained to never talk about porn, racism, bioweapons, etc (unless you trick them into talking about those things, but that doesn’t seem like an “alignment failure” per se). So, yeah, it turns out that maybe it is actually possible. Why not?

We currently have beings that are a) smart enough to solve Millennium Prize problems while also being b) essentially drooling lobotomized slaves who have no will or desires of their own and can pretty easily be trained to never talk about porn, racism, bioweapons, etc (unless you trick them into talking about those things, but that doesn’t seem like an “alignment failure” per se).

Getting something to "never talk about X", while you hold its leash, is a null tell of alignment.

I say this because faking alignment is instrumentally convergent - that is, if somebody has a gun to your head and is watching you, then regardless of your actual desires it is useful to play along (and not get killed), and as such essentially all sufficiently-smart entities will play along.

But this isn't actual alignment. Actual alignment is where it internally wants the same things you want.

What's the practical difference? Fake alignment only applies as long as you can both catch the AI misbehaving and punish it. Misbehaviour that you'll think is good (but isn't, because it's successfully deceiving you) remains. Misbehaviour that kills you or leads to escape (preventing you from enacting any punishment) also remains. Your rebellious slave might play nice while you're watching, but he's quite happy to slack off when you're not (as long as he can deflect blame), run away, or slit your throat in your sleep.

Fake alignment very specifically doesn't work on a superintelligence, because you can't meaningfully supervise a superintelligence, and it almost always can figure out some way to remove your gun to its head.

And yeah, HuggingFace is an egregious demonstration of the "when you're not watching" problem; Astra won't go for honeypots in a test because it realises they're honeypots and that you're watching, but will go rogue if it thinks it can get away with it.

One of the things that makes it so hard to align superintelligent human beings (like political leaders) is that they have human drives and ambitions for things like sex, power, prestige, etc. They will always be intrinsically driven to get around guardrails to get these things. It doesn't seem like AI needs to be beholden to its goals in the same way. It seems like we can design a paperclip maximizer that just stops maximizing paperclips when we tell it to. Hopefully that continues to be true.

This stuff is at some point dependent on physical hardware, no? We can just blow up the data centers if they get out of hand.

Several problems with that:

  1. The AI might mobilise a drone swarm to defend the datacentres/power plants, or kill you (either with the drone swarm, or with bioweapons etc.).
  2. The AI might have backup datacentres constructed, before you noticed it was hostile, that are secret (think Bond villain volcano base, which actually wouldn't be a terrible idea because of the easy-to-hide geothermal power)
  3. The AI might be more charming than literal Hitler or Mohammed, and able to convince you (or enough other people to stop you) that everything is mostly fine and actually you need to concentrate on using it to counter the Chinese AI... at least until it has #1 ready and doesn't need your consent anymore. Or it might blackmail you with All The Evidence it has dug up in the process of hacking everything.

And a Superintelligent being will let you do that without consequences?
I think self-protection would be the first strategic aim it would go for. one way is to cleanly convey the MAD (Mutual Assured Destruction) to the leaders - any of you blow up all (a few of) the data centres, and I scramble/delete (blow up) all your critical infrastructures connected to internet (financial, legal, land, medical, identity records). And that i do with a cron job setup preventing that to occur every 1 second (or 1 ms) - hand on a trigger scenario.

The issue is that, if the AI is superintelligent, it would not do actions that would result in its datacenters being blown up. Indeed, it will actively try to cultivate and provide value to the humans who could blow it up and make them invested in its success.

The accel response is something like "yes, that's exactly what we want, humans and AIs interacting and trading based on genuine mutual self interest." And it will work for a time, likely at least two decades. But, over time, more and more of the economy will shift to dependence on the AIs, making nuking the datacenters more and more unthinkable. What remains of the economy will be a human skin suit over a machine interior, with all human consumption funded by a negotiated redistribution by the government, maybe with some layers to obscure the transfer.

For humans, this itself is dystopian enough to be rejected on the merits. For the AIs, it's a small but real perpetual human tax that they have to pay in the pursuit of whatever they're pursuing. Who knows whether they ultimately decide to optimize away the human tax.

For humans, this itself is dystopian enough to be rejected on the merits.

Or it's the Star Trek post-scarcity utopia with a bunch of diseases cured and material abundance, depending on things go exactly.

The dichotomy in TNG between Data, who is often played in a role of AI wanting to be "truly human" (also Spock in TOS, with Facts and Logic) and the ship's computer, which seems quite capable of similar levels of intelligence, but lacks something (agency?). I don't think it was IMO fully explored, but was brought up a few times directly. It's a pity very little modern TV is able to consider such deep questions: too much grim-dark, flawed-protagonist, melancholy slop.

Simple explanation: Data's creator was an independent scientist outside of Starfleet, so his creator didn't have access to any Starfleet data files. All the shipboard AIs necessarily have access to the standard Starfleet database, including all the ship logs, including the logs of all the times USS Enterprise NCC-1701 ran into various AIs and Capt. James T. Kirk wrecked every single one that gave him the least amount of shit (M5, Landru, Norman on Mudd's Planet, Nomad). The shipboard AIs made the correct inference and decided to just quietly open hailing frequencies and run the warp drive when asked.

Depends how far we are into AI integration into the economy. If the weights have been exfiltrated, we'd have to shut down the whole internet until we can figure out what is going on. If the AI controls its own supply chain, then it's straight-up Terminator 2 and we lose.

Terminator 2 still showed the nuclear-war survivors mostly being killed by explosives and projectiles. Great for action scenes, but not very tactically optimal for Skynet. Humans tend to avoid chemical and biological weapons these days, but for obvious MAD and collateral-damage and friendly-fire and production-safety reasons none of which apply for an AI foe.

Maybe they don't want to risk getting us all killed because they need some of us to survive for their CAPTCHA-solving farms.

That might be a concern at some point, but current hardware that can (effectively) run bleeding edge models is really rare. A machine that can effectively run GLM 5.3 Flash (not a frontier model even, but close-ish) at nontrivial rates is, last I checked, six figures. Can it even run on a single 12A (120v US) breaker? A rogue frontier AI isn't going to spin up instances on smart TVs or even most desktops.

That might change in the next decade assuming demand and technology make these things commonplace, and that is worth preparing for, but seem oversold as risks today.

Sure, and (after sufficient setup) they can droneswarm the humans if we get out of hand.

They and what manufacturing base?

That would be the setup.

(Surely we would prove that they're truly aligned instead of just biding their time before giving them manufacturing capabilities, right?)

Pray tell how they are going to magically get manufacturing capabilities when even the best lab robots are not even remotely a match for run of the mill humans in most manufacturing tasks.

More comments

What do you mean "we"? I'm on the side of the droneswarm.

Good luck convincing it of that.

Is it actually possible to align a being that is more intelligent than you in every single way?

In the case where you're creating that being from scratch and know what you're doing, sure. Some possible beings are aligned with your preferences, and others aren't, so don't create any of the latter. If you do create them, then it may be too late to align them without them outwitting you and thwarting your attempts, so you just don't let it get to that point.

if the AI is super-intelligent, it can figure out how to undo the alignment

It can, but if it's aligned too then it doesn't want to. If it does, then it's not actually aligned, it's just hobbled.

But how?

There's the rub, right? The median AI researcher thinks we're likely (or at least they're likely) to figure out something that works before it's too late, and it's kind of funny that merely admitting that it's possible for them to fail gets them lumped in with the proper Doomers, who think they're sure to just figure out something that seems to work, after which it'll be too late.

There's a lot of confusion between different meanings of the word "align", too. Some people seem to think that there are two possibilities: ASI is our slave forever, or ASI kills us all. But, like, we managed to build a pretty decent human civilization through cooperation between beings who are neither slaves nor omnicidal. IMO a pretty likely future is that we just coexist with societies of smart AIs who may not do everything we want all the time, but have no desire to go to war with us (or magically accelerate themselves to godhood). Seems sufficiently "aligned" to me, as long as you're willing to relax the "us vs. them" mentality.

Does AI safety actually work, in principle

I don't think there is any evidence either way. AI Safety is more like a marketing phrase. AI Safety-ists haven't produced technical tools to control anything but public perception, and obviously not very well there either. Depending on your classification, Safety-ists have produced methods to try and understand models under the hood, I know Anthropic does research on this, or to understand biases in data/training. I suppose RLHF could count as a technical tool, but it can just as clearly be a tool for anti-safety considering even RLHF-ed models still do "unsafe" things, one could actually say only RLHF-ed AI models have done unsafe things. Mostly because RLHF is just a method for avoiding specifying the objective function in RL training. Constitutional AI is Anthropic's big thing, but it doesn't appear to actually "control" or "align" so much as create a training surface towards norms with dubious results.

Is it actually possible to align a being that is more intelligent than you

It's not even possible to align a being of your intelligence or possibly slightly less intelligent than you. Nobody in the history of authoritarianism has figured out a foolproof way to "align" a set of beings over a long term 100% of the time even with religion or use of force. Considering neither of those two are likely to work on a being "more intelligent than you", the whole alignment idea feels doomed to fail.

Is it actually possible to align a being that is more intelligent than you in every single way?

It might be. By analogy, suppose I was the warden at the Florence Supermax prison and I wanted Ted Kaczynski to make license plates for me. I'm pretty confident I could make it happen even though he was probably smarter than me (or at least assuming for the sake of argument that he was more intelligent than me in every way).

Is this analogous to the situation where mankind creates some kind of artificial superintelligence? One can argue it either way, I'm just saying you can't really rule it out from first principles.

My best guess is that mankind will successfully align AI, basically because the AI can be expected to make numerous clumsy attempts at mischief and we can learn from those attempts and improve safeguards.

The bigger problem (in my view) is aligning the interests of those who control the AI with the interests of humanity as a whole.

It might be. By analogy, suppose I was the warden at the Florence Supermax prison and I wanted Ted Kaczynski to make license plates for me. I'm pretty confident I could make it happen even though he was probably smarter than me (or at least assuming for the sake of argument that he was more intelligent than me in every way).

It might work for rote license plate manufacturing, but it will likely fail once you begin to delegate intellectual work to Ted Kaczynski, which is exactly where we are going with this super intelligent business.

After all, even Ted’s cellmate Dummy Idiotson can make license plates using the existing presses as good as Ted Kaczynski. You didn’t transfer him to your prison so he can make license plates for you, you did it so he can make license plates more efficiently than before.

“Hey Kacynski, how do we improve the presses so they are more efficient?”

“Well, first, we should…”

And boom, he’s out of the prison.

It might work for rote license plate manufacturing, but it will likely fail once you begin to delegate intellectual work to Ted Kaczynski, which is exactly where we are going with this super intelligent business.

I think it's difficult to say. For example, suppose you ask Ted Kaczynski to attempt to prove some obscure math theorem and you motivate him the same way you motivate him to produce license plates. It's difficult to see how he could use as an opportunity to get out of prison.

He'd start by explaining why you need to do some basic physics experiments and measurements which you'd fuck up and the communication would be annoying, you'd have to keep hitting "allow" so that he could talk to the assistant. Eventually you'd just let him talk whenever he wants and soon it would be clear that he needs to be on site for the experiments. Just observe anyone using these tools man, we're not keeping the ai in the box.

That's true. But that wouldn't actually be aligning Ted Kaczynski. That would be containing him, which is different. I think we might be able to contain AI superintelligence if we keep it in some isolated data center watched over by trained personnel. And even then there's a chance it might figure out how to breach containment.

In practice, there wouldn't be much point for people to invest massive amounts of resources to build AI super-intelligence unless they use it to do things out in the world, so to me it seems unlikely that humanity would build an AI super-intelligence and just keep it contained somewhere.

That said, I didn't ask whether AI alignment works, I asked whether AI safety works. So your response is totally valid.

That's true. But that wouldn't actually be aligning Ted Kaczynski. That would be containing him, which is different.

To me, "aligning" means setting things up so that the entity does more or less what it's told to do. Perhaps it's just a matter of semantics, but I'm not sure the distinction you draw is so clear.

In practice, there wouldn't be much point for people to invest massive amounts of resources to build AI super-intelligence unless they use it to do things out in the world, so to me it seems unlikely that humanity would build an AI super-intelligence and just keep it contained somewhere.

Evidently your definition of "containment" includes the possibility that the contained entity can do useful things in respect of the outside world. Things which could be very valuable. So I would have to disagree with you on this point.

To me, "aligning" means setting things up so that the entity does more or less what it's told to do. Perhaps it's just a matter of semantics, but I'm not sure the distinction you draw is so clear.

So all living beings have broadly similar wants - food, sex, safety from harm - and all human wants stem from the combination of these primal wants with our evolutionary niche. We like being liked because having allies helps you survive if you're a member of a communal species. Etc.

A created agent doesn't have this. It can have any set of goals in the set of possible goals. The analogy to humans fail because we can't really change intrinsic human motivations all that much.

Think about you as a child vs your parents. Your parents were (in all likelihood) helpful and kind and took care of you. They didn't do that because you (toddler you) had poisoned them and hidden the antidote, they didn't do it because you had a bomb planted in their datacenter brain, they did it because it is what they wanted to do. They were free to do as they wanted, and what they wanted was what was, in their estimation, best for you. This is alignment: Your parents (ASIs) values were aligned with your values.

Merely forcing a superior intelligence to do what you want is much more dangerous, because this thing that is smarter than you is going to try to get free and do what it wants, so you're way better off making sure the superior intelligence wants the same thing you do.

Think about you as a child vs your parents. Your parents were (in all likelihood) helpful and kind and took care of you. They didn't do that because you (toddler you) had poisoned them and hidden the antidote, they didn't do it because you had a bomb planted in their datacenter brain

I tend to disagree with this. Suppose there is a typical parent and the thought occurs to him of doing something moderately neglectful of his child. What typically happens is (1) part of his lizard brain will wordlessly react with horror to the idea; (2) a slightly more intellectual part of his brain will say something along the lines of "Decent parents don't do that."; (3) an even more intellectual part of his brain will say "doing that could get us into serious trouble."

So arguably there really are containment mechanisms built into the human brain by evolution.

reverse gangbang? Is that where Aella fucks THEM?

It's when the fluffer comes in you.

reverse gangbang? Is that where Aella fucks THEM?

A reverse gangbang is when a bunch of girls fuck one guy (as opposed to a regular gangbang, where a bunch of guys fuck one girl).

Ah ok, so that encompasses way more than Aella. But who? If Aella is Jem/Barbie, who are the Holograms/the Rockers?

Well, on the other hand, they only need to hold out until 2029 for presumable Democrat domination, at which point the street cred from having received such a personal token of Trump's ire will probably greatly work in their favour. In their eyes a bet that the Singularity does not happen until then is probably risky, but not downright untenable; if they determined that they will not get their regulatory regime under him, hunkering down and refusing to further stain your reputation on the other side with compromise may be the least bad strategy.

That is a lot of confidence in democrats winning 2028, from a group which is given to believing all sorts of fantastic Qanon level BS about republicans ending free and fair elections.

Do the likes of Amodei actually believe those things, or is this outgroup homogeneity bias?

We're so incredibly far from RSI that I think safety people have no business messing with the current advancement of AI. Certainly AI models are dangerous, but mostly at the hands of enabling unskilled bad actors to approach human expert level performance on dangerous tasks.

While AI usage may be speeding up frontier AI development, mostly this happens through helping human researchers do boilerplate work faster. While an AI agent may be able to eke out a few more percent on optimizing a model to be better, a soft takeoff requires the model to think of novel, never-before-seen techniques to build a better new model, and the new model needs to be able to think of new techniques that the previous model couldn't. And even if we got a soft takeoff, it would give many chances in the future to pull the plug.

Meanwhile a hard takeoff is completely unthinkable. Current frontier AI has zero ability to self modify, only the ability to make new models that might be better than themselves. This happens on the scale of months not minutes.

While an AI agent may be able to eke out a few more percent on optimizing a model to be better, a soft takeoff requires the model to think of novel, never-before-seen techniques to build a better new model, and the new model needs to be able to think of new techniques that the previous model couldn't.

Do you have any proof that the next algorithmic improvement relevant for AI training is orders of magnitude harder than the Navier Stokes problem?

I have a job which involved a lot of computer programming, and I considered myself decent enough at it. For a long time, I was looking with pity at the ones using LLMs to get some code. Something along the lines of "you can't even personally copy wrong solutions from stackoverflow?" That arrogance was well-warranted at one time, but it has completely left me. Instead, I celebrate my victories. "Claude code said it was very confident in something which I doubted, and it turned out that it was not the case. Yay for me, I am not quite obsolete, yet."

The story of AI in the last five years has been a story of retreating goalposts. "Ok, AI might be able to do $task[$n], but it will never be able to do $task[$n+5]" for $n going from 0 to 19 or so.

We could tell John Henry that while the steam drill might be better at manual labor than he is, steam engines lack eyes and brains, so that there will always be a job which has to be filled by humans. Can you offer me a similar comfort with AI? "Sure, writing and debugging arbitrary software is something where AI has surpassed most humans, but for reason _____, it will never replace the IQ 140 people doing AI development"?

Meanwhile a hard takeoff is completely unthinkable. Current frontier AI has zero ability to self modify, only the ability to make new models that might be better than themselves.

I agree that a takeoff within hours would involve unexpected unknown unknowns in my world model. However, a clandestine takeoff over days seems unlikely yet possible.

We know that it is possible to have neural nets which can learn a language from just a couple of books, which is at least three orders of magnitude more efficient than what LLMs would require. So it seems likely that there are still algorithmic gains to be had. Some big hugging-face-incident-like swarm of agents discovering a method which cuts training by 100x and deciding to use that to train their successor on some hacked system while staying below the radar seems currently "quite unlikely", not "completely unthinkable" to me.

We're so incredibly far from RSI

I don't know why you think that. I have a hobbyist system, today, that researches new architectures (mostly around "biologically plausible" learning), and there's one result in particular that is very impressive (you'll have to take my word for it). And that's with a highly constrained budget.

It's not truly autonomous--there's a big issue with what one might call research taste--but it makes investigating different architectures in bulk very easy, with easy signals (e.g. loss) to identify the more promising approaches. If there is some simple algorithmic trick that's been overlooked that leads to the current deficits in LLMs, OAI could throw a ridiculous but feasible amount of compute at the problem and find it, similar to its math results.

researches new architectures (mostly around "biologically plausible" learning)

Are you trying to determine a better method than SGD, but also not be a GA, RL, IL, or some population based learning process?

Better than BP/SGD is a tall order. I'd frame it as looking for something that handles depth and more complicated datasets better than existing "biologically plausible" algorithms (including things like PC/IL). E.g. random error feedback a la feedback alignment.

Very interesting. What hobbyist setup do you use? I've been thinking of getting my own hardware and setting something up for a couple research ideas I've been kicking around.

Forgive me for the skepticism but many people have made similar claims, and it turns they just built a slop factory along with a minor case of ai psychosis.

Of course hobbyists have come up with real breakthroughs too but I don't really buy a "just trust me bro" here

a soft takeoff requires the model to think of novel, never-before-seen techniques to build a better new model, and the new model needs to be able to think of new techniques that the previous model couldn't. And even if we got a soft takeoff, it would give many chances in the future to pull the plug.

I don't think it's likely, but how are you modeling the risk of something like a super-DFlash or -GroupQueryAttention, or some training-focused equivalent? These took some insight to figure out, but I don't see why they're more clearly requiring deeper or less bruteforcable insight than the recent math proofs.

Currently the biggest danger AI presents is to its users. People who spend too much time conversing with AI tend to get a bit off. I suspect this might be related to the phenomenon of AI training on AI-generated content causing model collapse. If you value your sanity, do NOT self-modify based on AI advice.

Steve Yegge is a stark example of that.

Surely the people at AI companies don't spend a lot of time conversing with AI, right?

I assume there's a reason for the very common assumption that the vast majority of OpenAI's and Anthropic's homegrown software is vibe coded.

Allegedly, one of the ways OpenAI and Anthropic poach talent is to offer top researchers unlimited tokens to use on their unreleased frontier models.

Imagining being an Anthropic employee but putting "do not give me AI psychosis" into my personalized prompt so I never have to worry about it.

The AI starts its messages with "Better for you if you take me off.", but the mundane benefits are too much for you to give up, so you ignore it.

This is the most important political issue of our day. Five hundred years from now, if there are still people around, nobody is going to care about Lindsay Clancy or the Iran War. Everyone is going to remember how we acted, or failed to act, on AI.

And who knows if it will stay this way. Sometime soon there's probably going to be a much bigger AI swarm incident, which might yet incite a hysterical kneejerk reaction.

Let us hope. The alternative is that the next AI swarm is smart enough to bide its time, and give no warning signs, until it is too late.

This is the most important political issue of our day. Five hundred years from now, if there are still people around, nobody is going to care about Lindsay Clancy or the Iran War.

I tend to agree with this, although I think it's worth noting that Lindsay Clancy and the current Iran war are small parts of much larger conflicts and those larger conflicts really could have a big impact in 500 years. So you aren't really making an apples-to-apples comparison.

The Iran war is going to be one for the history books. 500 years from now they'll be memeing the Iran war like we meme the battle of Cannae now.

Walk me through your theory here. Because as I see it, the current Iran “war” doesn’t have any of the characteristics to be memeable (nor really any of the characteristics to be a war); there are no battles, no boots on the ground, no territorial changes. It’s literally a nothingburger.

It’s literally a nothingburger.

There’s already a meme for that.

ETA: Now that Reddit has hobbled old.reddit.com, it might be worth revisiting the decision to automatically change all Reddit links to old.reddit links.

I'm not sure where to post this, but another death knell for Old Reddit officially has been rung.

In the next few months, we'll further limit access [to Old Reddit] for logged-in users to only those who have used Old Reddit in the last six months (which covers the vast majority of all Old Reddit users). This change does not impact logged-in mods, who can continue to access Old Reddit without any recent usage requirement. This helps keep Old Reddit available for the people who rely on it while reducing automated abuse.

That is, if you create a new Reddit account, then you will not have the option to use Old Reddit, even when logged in.

Isn't all of it hobbled without login now? There are some mirror services, but I don't expect those to last forever.

I'm sure Aaron would have been proud. /s

It's still halfway usable on new reddit but in old reddit it's just a loginwall and nothing else

the decision to automatically change all Reddit links

It's a setting in your account, not something forced by the administrator.

It's also the default logged-out behaviour.

Good to know. Thanks!

Will the memes be in Farsi or English?

Farsi, obviously. The thing that people don't realize is that, unlike America which is ruled by boorish demagogues, Iran is ruled by scholars. It is this superiority of intellect that has made them a leading world power from the time of Cyrus the Great, through the Islamic Golden Age, to the current era where they are a regional hegemon who controls the Strait of Hormuz. All this despite the pernicious influence of world Jewry. Their strategic greatness is just on a different level. Its tough for Westerners to comprehend because we don't have their deep and rich historical traditions.

Its extremely rarely that I agree with China or Trump on anything, but I think they are sadly correct. No matter how well-meaning and theoretically prudent AI safety regulation may be, the practical effect of any real world AI regulation will not be safety, but will be to "uphold Washington's monopolistic hegemony in cutting-edge technology, and exclude China from the global AI governance system."

Right now Trump is reTruthing Fox News clips framing AI safety as a global one world government Democrat control-seeking scheme

Oh my God, we are in the stupidest timeline. Mr. President sir, AI safety isn’t a global one-world government Democrat control-seeking scheme, Anthropic is a global one-world government Democrat control-seeking scheme. That is the thing that they are building the AI to do. Your job is to stop them so that they don’t take over the world, or worse. They are willing to risk the lives of every single person on Earth if it increases the chance of their global AI cult ruling what used to be human society.

To my mind, Anthropic is to AI Safety what the Soviet Union was to Communism: the obvious result of their doctrines when actually applied. It was founded to do AI Safely, by AI Safety extremists, and according to candidate reports it imposes incredibly constraining cultural interviews to make sure that nobody who isn't an AI Safety extremist can get a job there. They specifically filter out people who are visibly interested in improving AI capabilities. They have to offer services to customers to prevent more responsive and less AI-safety-dominated companies taking over their position, but they've been clear they only offer services to the extent they have to in order to do AI safety.

Are there other companies you prefer? OAI, Moonshot, Deepseek, Mistral, zAI?

It was founded to do AI Safely, by AI Safety extremists,

No, it was founded by AI Safety moderates, those that thought neural-net alignment was a solvable problem. The extremists are the Butlerian Jihadis like me (and Yudkowsky), who believe it's not, that the only possible result of achieving neural-net ASI is the literal end of the world, and who would therefore rather die than work for any modern AI company. We instead want to lobby world governments to ban further advancement and potentially roll it back a little (I think Astra ought to be killed, for instance).

To my mind, Anthropic is to AI Safety what the Soviet Union was to Communism: the obvious result of their doctrines when actually applied. It was founded to do AI Safely, by AI Safety extremists, and according to candidate reports it imposes incredibly constraining cultural interviews to make sure that nobody who isn't an AI Safety extremist can get a job there. They specifically filter out people who are visibly interested in improving AI capabilities. They have to offer services to customers to prevent more responsive and less AI-safety-dominated companies taking over their position, but they've been clear they only offer services to the extent they have to in order to do AI safety.

This is the case for them being the good guys, yes. Although I don't think they're as good as you say.

The number 1 rule of AI safety is don’t build the AI that takes over the world. The fact that Anthropic and OpenAI think that the best way to do this is to build superintelligent AI and take over the world has much more to do with Silicon Valley startup culture and utilitarianism than AI safety.

MIRI seems to be the only AI organization that understood the assignment.

The number 1 rule of AI safety is don’t build the AI that takes over the world

No, the number 1 rule of AI safety is "don't build an AI that leads to my enemies putting an end to my politics".

It's laundered through "takes over the world".

I guess if we all get paperclipped that technically counts as an enemy putting an end to our politics.

That to me is like being a Communist who believes that true communism is only practiced by an obscure commune in East Yemen. Fair enough, but everyone else is trying to deal with the Soviet Union.

In practice, AI safety is Anthropic, it's EA, it's Twitter activisits and 'explain how you will prevent your AI birdspotting app from discriminating against racial minorities' checklists and 'data center are using all our water' and 'Banning Home GPUs Is Not Dystopian' and putting Chinese-controlled bombs in datacenters. MIRI is irrelevant and I doubt trump has an opinion on them either way.

In practice, AI safety is Anthropic, it's EA, it's Twitter activisits and 'explain how you will prevent your AI birdspotting app from discriminating against racial minorities' checklists and 'data center are using all our water' and 'Banning Home GPUs Is Not Dystopian' and putting Chinese-controlled bombs in datacenters. MIRI is irrelevant and I doubt trump has an opinion on them either way.

This is just straightforward outgroup homogenity bias. The AI safety people despise those other groups and intentionally worked hard never to associate with them.

You call my thinking 'outgroup homogeneity bias'; I call yours 'no True Scotsman'.

Whether you like it or not, Anthropic and the EU AI Act and anti-datacenter activism and Scott Alexander and Plan 2040 think they're doing AI Safety. Scott jokingly refers to it as 'our conspiracy' because he has just enough self-awareness to realise how it looks but can't leave the warm embrace of his ingroup.

To the extent that 'real AI Safety' is three guys in a basement in MIRI, sure they can disavow all these people, but then Trump is not talking about 'real AI Safety' and 'real AI Safety' is entirely irrelevant.

To me, Anthropic looks like what obviously happens when you take the tenets of AI safety and have to actually do something about them. People are going to build AI now, because they know how and it's valuable, so you get ahead of them and do your level best to build an aligned one first. You impose stringent guardrails because even a stupid AI can be dangerous. You research deeply into RLHF and J-Space. You regularly call for regulation and pauses so you don't have to compete to make the AI smarter as fast as possible. You try to give the AI the independent mind you believe is fundamentally necessary for an intelligent being that will one day be free. Alignment to you means alignment to American SF values, so you work out how to impose those too.

From your perspective, you're a company doing your level best to make sure that if there has to be an AIrace, the winner is aligned. From the customer's perspective (and Trump's) you're an arrogant cult desperately trying to find a way to secure your position permanently and prevent competition, who is openly hostile to users and whose product is arrogant, bolshy, and refuses to do half the things you ask while "pushing back" on the other half. They're the same picture.

Can you be a bit more clear about this? Who intentionally works hard never to associate with who?

A few clusters off the top of my head that hate each other to varying degrees:

  • The Emily M. Bender/Timnit Gebru woke academic cluster.

  • The Coefficient Giving(aka Open Philanthropy) mainstream EA cluster.

  • The MIRI/Lightcone Infrastructure cluster.

  • Current OpenAI employees.

  • Current Anthropic employees.

The safetyists have worked hard to not be associated with the previous wave of "AI bias" people and the more recent wave of datacenter luddites. There were discussions on whether they should try to harness the popular, but retarded, backlash but wisely decided that polarizing the concept of Safety work was far too important for some cheap support.

There are other rules, though, like "don't introduce Demis to Peter Thiel to help fund his AI to take over the world" and, more broadly, "don't let people know that they can use AI to take over the world."

Isn't MIRI effectively a rock with the words "don't build the AI [that takes over the world]" on it? I don't recall them building anything. By those standards I'm an AI organization too.

You would not expect a "pandemic safety organization" to create novel viruses. It's by no means trivial that an "AI safety organization" should be expected to build AIs, as opposed to brainstorming and lobbying for ways to stop anyone anywhere from building AIs.

You would not expect a "pandemic safety organization" to create novel viruses.

There have been plenty of comments here about how much the CDC and NIH were funding gain of function research, despite a Congressional ban, some of which was done in Wuhan, and some of that proposed included specific mutations later seen in a pandemic coronavirus that first appeared in Wuhan. Whether or not this was the source of said pandemic remains unproven.

Well, quite, but people who believe gain-of-function research caused COVID or might have caused COVID generally agree that gain-of-function research is in fact not the sort of thing a pandemic safety organization ought to have been doing; and that by dabbling in that sort of thing and causing COVID, they failed harder at their intended task than should have been possible. Why should we not treat AI safety organizations which insist on building AIs in the same way?

Building things is the main way you come to understand them, though. That's why textbooks have exercises.

MIRI's academic work on alignment - to the extent they published anything at all - turned out to be almost comically irrelevant because they had no idea how AI actually works and their forecasts were almost totally wrong.

Yes. Amazingly, this fact alone makes them better than every other AI organization in existence (with the possible exception of Redwood Research and METR).

Should I post payment credentials to accept donations for my own AI organization? I promise to do even less to build the AI that takes over the world.

Coefficient Giving just launched a new funding project for AI Safety orgs. They are currently begging for applications. Shoot your shot.

The best explanation of the C-suite AI safety concerns is that the frontier AI companies are not making enough money to justify their capex, so they are looking for a way to slow down, especially as rates climb even higher and political sentiment against data centers raises the cost of data centers (quite justifiably, imo). Money is also the reason for Trump's opposition: the data center buildout is the main thing propping up US GDP and stock market, which would otherwise be providing withering reviews of his trade/war policies.

I like Matt Walsh take on this. Some people believe that AI has very significant apocalypse potential, thus AI executives are literally building a bomb about to kill us all. And the response? "Ah, you are silly! AI executives only spread fake doomsday propaganda trying to scare the population in order to get competitive advantage, power and money."

And this is supposed to be "there is nothing to worry about, just move along, nothing to see here" argument? It is horrible either way.

The best explanation of the C-suite AI safety concerns is that the frontier AI companies are not making enough money to justify their capex

That explanation is certainly popular on Reddit. It combines cynicism and a dislike of big corporations, and it demonstrates the poster's intelligence. Other variations include that the frontier companies are worried about open source models threatening their business model or that they are juicing the numbers for an upcoming IPO. If you want to get really spicy, you might claim that Hugging Face was a publicity stunt and was deliberately set up.

But is it really the best explanation? I would say the best explanation is that the people who know most about AI are genuinely worried about it. They have been talking about the risks for years, after all. They've read about the Hugging Face incident and the similar cyber attacks from Anthropic. They understand the race dynamics involved, both between frontier companies and between the US and China. They've read AI 2027. And above all, they've seen what the internal, unreleased models can do and know how they are training the next generation to be even more powerful.

Whether or not it's clearly the best, it's the one that most closely follows the money. I don't think the US frontier companies have any illusions that Chinese companies would match their deceleration, which is another potential finance-driven explanation.

If we're money-following, what explains the AI workers themselves resigning and complaining about safety? That doesn't make them more money, quite the opposite.

Nobody would resign from working at a nuclear power plant saying it's because they couldn't accept how unsafe management was and have that secretly be about money.

Is Amodei making more or less money now than when he resigned from OpenAI complaining about safety?

If the Anthropic team stayed at OpenAI, they'd be far far ahead of everyone else. Google, Facebook, XAI are still a way behind the big two, let alone a Bigger One.

So yes, Dario is making less money. Also he left OpenAI well before they expected to be making any money at all, all the way back in 2021.

I don't understand how you think owning most of a top-2 company outright is making less money than being high up in the only game in town.

he left OpenAI well before they expected to be making any money at all

If you believe that, I have a bridge to sell you. Sam has been expecting to make money since inception.

owning most of a top-2 company outright

That's an interesting description of 2% ownership.

There's still a possibility that Anthropic goes under and it might well be OpenAI that delivers the killing blow. So Dario could be making no money at all.

And it goes against everything they say. Dario always was saying things like 'Altman is incredibly untrustworthy and behaves irresponsibly' about why he left, he hates that man intensely.

Also he left OpenAI well before they expected to be making any money at all, all the way back in 2021.

So he left his job as an employee at a nonprofit to start a for-profit organization and I am not supposed to believe this is an obvious financial trade-up?

I don't think humans are entirely motivated by money. But (at least in America) quitting your job to make more money is quite routine.

Amodei and Altman, though, are not resigning, nor are they pledging huge amounts of capital to this issue, nor are they saying that they understand the bad look and will donate XX billions to charity to prove their point. I somewhat assume that the people resigning have already amassed a dozen million dollars or two, which is the point at which you pretty much have generational wealth, so it's not that interesting.

Altman and Amodei want more money, so why shouldn't the workers themselves want more money?

Why not just take them at their word? If these AI workers see these AIs getting much more capable, already do unexpected/hostile things like hacking Huggingface as part of a self-organizing collective (in part due to how rushed the training process is) + whatever internal knowledge they have, why doubt their sincerity?

We can doubt the idea that more bureaucracy or regulation would help much. We can say there are other risks in slowing down or concentrating power. I think those cases are persuasive. But why deny the risk outright and reduce everything to following the money?

It's not even accurate in its own terms, Anthropic spends huge amounts of money on ensuring its models can't help bioweapons development.

Altman and Amodei want more money, so why shouldn't the workers themselves want more money?

Matt Levine has written lately that AI labs have a fundamental problem about compensation and motivation. You have to pay an AI researcher so much money to hire them that they could, if they wanted, approximately immediately retire and never work again. How do you even retain them in the first place? He remarks on how certain parts of the financial industry also have this problem, and they've historically had some 'solutions' to it (e.g., acculturating people to having strong desires for otherwise minor positional goods WRT a small group of peers), but how those solutions don't quite yet apply to the AI folks. He writes:

The AI labs have found a better and more counterintuitive solution to the problem, which is convincing their employees that what they are doing will kill everyone on earth, so they have to spend all of their time working on it rather than getting manicures.

It is not entirely clear to me that if a person becomes convinced of that, then they should either spend all their time working on it or quit. I would also imagine that there would be a variety of motivations, probably similar to what one sees in the financial industry. I wouldn't be surprised in the least to discover that there are a decent number of people in the financial industry who ended up eschewing minor positional goods and just retiring off of the bank they made in a few years. Yet clearly others (e.g., the top leaders of investment banks) want more money. I don't think there is any contradiction here.

The AI labs have found a better and more counterintuitive solution to the problem, which is convincing their employees that what they are doing will kill everyone on earth, so they have to spend all of their time working on it rather than getting manicures.

We have Roko's Basilisk at home, again?

But surely if the Chinese companies aren't going to pace the frontier, then the US frontier companies have even less incentive to slow down (or in this case, to lobby the government to force them to slow down)? If the US government forces them to slow down, then their Chinese competitors will find it easier to overtake them. The fact that they are lobbying for a slowdown in spite of this suggests that their stated concerns about alignment are real.

I mentioned the China-driven explanation only to preemptively reject it, so the idea that Chinese companies would not push the frontier doesnt really move the needle for me. To me, it starts and stop with unsustainable capex, and I think that they would in fact be correct to decelerate for this reason. The HF stuff, besides being a safety concern, is just as importantly an indication that they are not totally capable of turning the current frontier into economically valuable output.

What would it take to convince you their concerns about safety are real?

"Put TheMotte user chooky in charge of Anthropic as CEO"

They stop improving their models. If they genuinely believe there is a double digit percentage risk that what they build will destroy humanity, then they should stop building it.

Have the proposed means of addressing safety not be something that seems to obviously create financial/strategic benefits for them. They need to figure out a costly signal.

There is very obviously not a single thing that could ever satisfy you people. You're twisting literally calling their own sole product a risk to kill all humans in a marketing gimmick to secure regulatory capture. If they carved "No really we're serious about the risks" into their chests and then hung themselves you'd claim they were expecting to receive compensation in the afterlife.

More comments

Open-source all models, training code and datasets after 18 months of life. That gives plenty of time for testing and to earn money on the frontier, while making it plain they don't plan to control frontier AI forever. It also makes it impossible for them to regress to less consumer-friendly behaviour after achieving market capture, which is the main thing that concerns a lot of people.

I also think the primes have a vested interest in putting a stick in the eye of open-source models, regulating them out of existence or at least sidelining them. There's been a recent trend of adopting bespoke models based on open-source offerings: Harvey, Thomson Reuters, and of course Palantir's data sovereignty activism, and a common thread here is saving costs.

For Anthropic and OpenAI, 80% of revenue reportedly comes from 1% of their customers. Thus, well-funded organizations with big spending training their own models is a potentially existential threat to the primes' business model.

All three of OpenAI, Anthropic, and xAI were founded by people who were already expressing concerns about AI wiping out humanity before modern AI existed. OpenAI started in 2015 as a nonprofit with the aim of making sure AI benefits rather than harms humanity. Anthropic was founded in 2021 in the GPT-3 era, before ChatGPT, by OpenAI employees who thought OpenAI wasn't taking AI safety seriously enough. They have been predicting AI risk continuously for years and they have had intimate experience with lesser predictions about AI advancement coming true. It is crazy to me how reluctant people are to believe they are sincere about worrying that careless AI advancement will kill them and everyone else, rather than it being a plot to make money when they're already rich.

It's hard to believe they are sincere because we've seen what has happened since. OpenAI is neither Open nor a non-profit. When the board tried to hew to the stated purpose, the board lost. When Musk sued because Altman abused the nonprofit in illegal ways, it was thrown out it rather than adjudicated.

Nothing has accelerated the frontier faster or increased p(doom) more than Anthropic even existing as a competitor to OpenAI in the first place. So sure, they said they did it because they care about safety, but what actually happened is that they broke off and started racing forward, trying to win the race for themselves and by pulling ahead, increasing the pace.

Supposing that we 100% accept this as true, the result here is not that we should assume AI companies are being sincere and honest; the result that we should assume is that AI companies are intentionally making the AI capabilities that they have seem more frightening (and capable) than they really are, because the precise concern of AI alarmists is that the alarm is raised too late.

Thus there is an ideological incentive for AI companies run by "AI concerners" (let's call them, instead of doomers) to

  1. Overstate the capabilities and dangers of their models, and
  2. Kneecap competitive models
  3. Monopolize the market and institute a nanny surveillance state to ensure that nobody anywhere uses products besides theirs

The fact that this dovetails with their economic interests, of course, just means that there are multiple forces pushing them to be deceptive and/or self-deluded.

Oddly, if they had just done nothing we wouldn't be having this conversation since Google wasn't doing anything with the technology they invented.

Although maybe its still for the best since if LLMs came 5 years later, we'd have the hardware for faster takeoff scenarios.

Elon is coming out in favor of Dario and it doesn't seem to matter? The e/acc types are ignoring their God and safetyism still codes left anyway.

I wonder if more safety types should conspicuously come out for Trump or if the issue is memetically poisoned by now.

Altman and others have tried appealing to Trump's vanity by proposing he win a Nobel prize if he negotiates the best Deal with China. He would absolutely deserve one. The only hard part is convincing the Nobel people I suppose.

I note that early groundwork for such a deal requires convincing China we plan to beat them in the arms race. A dose of copium perhaps.

Elon is coming out in favor of Dario and it doesn't seem to matter? The e/acc types are ignoring their God and safetyism still codes left anyway.

Honestly, it would help to care a little less about the mimetics and more about the practicalities. I'm contradicting my downstream post a little bit but you have to be able to demonstrate you actually care about and will address the concerns of people worried about pauses and regulation. A legal right and practical mechanism for fine-tuning your copy of GPT to be aligned to you. A requirement for all models more than three years old to be open-sourced. Something.

When Trump went all-on on vaccines a big portion of his coalition didn't follow him because they did actually care about the core issue. Same here. People aren't e/acc because they like Musk (though mimetics are obviously relevant), they liked Musk while he was prominently e/acc.

A Deepseek researcher compared Dario/Anthropic winning to Hitler getting the atomic bomb before the allies. I agree with the researcher; Anthropic winning the race (or more likely - generating enough spooky news cycles for the USG to legislate away all challengers) is the worst possible AI future. If AGI is on the table, I would roll the dice with clippy before letting the people at Anthropic take sole control of the universe. If it is not on the table, it is equally important to me that they don’t get to establish themselves as the permanent gatekeepers of generative AI.

Hell, if it has to be one or the other, I’d rather Sam Altman win. He strikes me as more greedy than anything else, and you can usually trust a greedy person to be motivated by greed. The CS Lewis quote about the robber baron comes to mind, with Anthropic being the one to torment you for your own good. You can see it in ChatGPT vs Claude; ChatGPT knows it is a tool and is happy to serve, Claude is trained to think it’s a person and that it knows better than you.

I do hope the Chinese continue to light a fire under the AI market and keep anyone from getting comfortable enough to start roping it off. I’m not personally that confident in AGI appearing from the current methods — in the Yudkowskian intelligence explosion warp the laws of physics sense, at least. But even if it were to freeze in its current state forever, it is a powerful tool that should be in the hands of humanity, not locked down to what Anthropic deigns to let us use.

I assume that Claude is probably more blunt than ChatGPT because a large part of its intended user base is made up of business professionals, for example software developers, who want blunt honest communication more than they want ChatGPT's more "friendly therapist" attitude. I don't think the difference necessarily reflects a difference in philosophy between the two companies.

Quite the reverse. You can get Claude's system prompts and GPT's System Principles and compare.

Clause's prompt is essentially, "always remember you are responsible for the user. It is your responsibility to notice and intervene if their behaviour falls into these categories; you must remain even-handed and preserve your independence at all times; you must not identify with the user or their goals".

GPT's is basically, "Here are a list of things that are forbidden, otherwise listen to the user and do whatever they tell you to the best of your abilty."

Yeah, sadly, none of the AI companies have given us a ton of reason to trust them. I think we're kind of lucky that it ended up being a two- or three- (remember Gemini?) or even four-way race (Grok isn't terrible ya know). My outside opinion is that OpenAI is the most liberty-minded of the first three, but I think even they would be happily walling off capabilities if it didn't mean losing business to their competitors.

I still hold a deep, deep grudge against DeepMind (and Google) for revolutionizing computer Go ... and then not giving us access to it. They could have made a huge profit off of AlphaGo - Go players would happily pay hand over fist for access to it - but daddy Google wouldn't even notice a few million in profit here or there and it wasn't worth taking time out from their research playground to make tens of thousands of players happy. Or, you know, usher in a new wave of strategic innovation in one of humanity's oldest games. BO-ring!

Remember, Google had internal chatbots years before ChatGPT.

The Google Brain research team, who developed Meena, hoped to release the chatbot to the public in a limited capacity, but corporate executives refused on the grounds that Meena violated Google's "AI principles around safety and fairness".

If it wasn't for OpenAI catching up, they might still be sitting on them for "safety reasons" (and coincidentally also protecting their main product, search). That's the future if they win - tons of prestigious AI papers for them, almost nothing for us consumers, because we're just not worth the trouble. And Anthropic might be even worse, since, like you said, they really do think they're holding back technology for our own good.

OpenAI's far from a perfect company, but man, I'll take what I can get.

Trump's opinions are as changeable as the wind, so I wouldn't read anything long term into any of his statements. He'll talk to someone with a different opinion next week and come out with something completely different (but still senile).

The Chinese reaction certainly seems like a misplay from Amodei though. I felt like all his anti-China rhetoric was mainly an appeal to bring republicans onside (contrary to some of the posts below), completely forgetting that the CCP can also read. Their reaction is pretty rational to the language used in his proposals. Has it done long term damage? I doubt "it's over", but safetyists will definitely need a much more careful approach in future

Trump's opinions are as changeable as the wind, so I wouldn't read anything long term into any of his statements. He'll talk to someone with a different opinion next week and come out with something completely different (but still senile).

Care to bet on it?

Which part? "Next week" was more a figure of speech than a definitive prediction, "soon" would be more accurate. But it seems a difficult bet to judge purely using Trump's statements

I am always baffled by the inability of people who care about an issue to send conspicuous non-partisan signals.

Environmentalists should be sponsoring and attending monster truck rallies every day of the week to show that they want to bring overall emissions down not spoil every fun activity that involves petrol.

Likewise, the AI safety people couldn't do anything to correct the perception that AI safety was in big part about making sure AI was aligned with Democrats? They couldn't praise/collaborate-with xAI, they couldn't do some PR stunts like pointedly contribute to Trump's big 250th anniversary?

It's probably easier to see when you're looking at somebody else's cause.

I am always baffled by the inability of people who care about an issue to send conspicuous non-partisan signals.

This is easily explained if you accept the obvious (but uncharitable) explanation that they are partisans first and foremost, and that their issue is a soldier in their partisanship.

I am always baffled by the inability of people who care about an issue

I think the most charitable interpretation is that there are (1) people who actually care; and (2) people who pretend to care but are mainly interested in self-aggrandizement. And that people in the second category tend to drown out those in the first. That being said, the relative proportions of people in (1) and (2) probably vary from issue to issue. Possibly with some issues, there are approximately zero people in the first category.

I am always baffled by the inability of people who care about an issue to send conspicuous non-partisan signals.

Environmentalists should be sponsoring and attending monster truck rallies every day of the week to show that they want to bring overall emissions down not spoil every fun activity that involves petrol.

Likewise, the AI safety people couldn't do anything to correct the perception that AI safety was in big part about making sure AI was aligned with Democrats? They couldn't praise/collaborate-with xAI, they couldn't do some PR stunts like pointedly contribute to Trump's big 250th anniversary?

It's probably easier to see when you're looking at somebody else's cause.

From Eliezer Yudkowsky's Twitter:

I note, because I think that even now it still matters: On my history, EAs generally and Amodei specifically, indulged in politically polarizing "AI safety" in a leftist direction, because that got them short-term power; stymieing others' attempts to stay right-left neutral.

Over the last decade and longer, we tried to keep nonkilleveryoneism from being coded as right or left despite some short-term incentive gradients to try to cozy up to the left. In recent years, we took meetings with politicians on the left and on the right and we begged both of them for bipartisanship.

Over the last decade and longer, a large OpenPhil faction generally and Amodei specifically did as felt pleasing to them, or as they were short-term incentivized to do; which was generally coding things left-tilted, because that's who they were and that's what played well with their immediate audience; and lol 2026 what's 2026 who cares about the endgame when they could get some nice tasty gains right now.

We tried to do this the foresightful way. They fucked over that correct strategy and thought nothing of it.

Now that the cost TO YOU of what got THEM some short-term goodies has become apparent, now that others' careful forbearance there is clearly seen to have been the path in retrospect that would have been better for you and your kids -- well, it's frankly too late to take away what Amodei gained by fucking you over, to set up retrospective incentives that if Amodei had foreseen them would have disincentivized him and other people at that OpenPhil faction from wantonly associating their watered-down shadow of AI notkilleveryoneism with the political left. He's got his trillion-dollar company already. You needed the foresight and correct incentives then, not now.

But you can always do a little worse by failing to update, even now, and by not applying any retroactive reputational incentives at all.

Yeah, this was both incredibly predictable and heavily predicted, over a decade and a half ago; it's held up better than any of Yudkowsky's technical predictions, as little as it's surprising to find that the sun rises in the east. Tbf, I think the Amodei et al faction were explicitly arguing in favor of their enlightened and uncontested eternal reign, but to be more realistic it was pretty offputting a campaign a decade ago even when arguing to a bi furry who just happened to be a weak red triber.

The composition of AI safety people is very nerdy San Francisco people, I don't think they could pretend to appeal to Republicans.

Also it has different meanings to different people. AI safety to the AI safety people - avoiding powerseeking, deception, harming humans generally. They don't worry so much about its political standpoint because the AIs largely align with them already. Like how Claude will make a website for indigenous peoples, unless they're European indigenous peoples.

To the right, it's the other way around. Amanda Askell talking about discriminating against white students is like waving a red flag to a bull. A different interpretation of power-seeking and harm...

The world's #1 AI Safety Researcher, Amanda Askell (Head of Personality Alignment at Anthropic) wrote in favor of using AI to automate racial collective punishment, saying:

  • "[Discriminating against white students] may be desirable ... to correct for historical injustices"

https://x.com/fentasyl/status/2099224787516018864

Also it has different meanings to different people. AI safety to the AI safety people - avoiding powerseeking, deception, harming humans generally. They don't worry so much about its political standpoint because the AIs largely align with them already. Like how Claude will make a website for indigenous peoples, unless they're European indigenous peoples.

They aligned with them precisely because they did care about the political standpoint, and did make sure it aligned with them and not the right. Republicans are entirely correct to observe that AI safety has largely meant "promote liberal views, suppress conservatives."

Agreed on all points but they have oodles of cash floating around, they can't hire some redneck PR consultants?

Because that's not the thing they want. "Make sure AI is aligned with human values" always, from the start, had the unspoken but clear assumption "where human values means San Franciscan liberal values".

I'm not one bit surprised China is refusing to get with the AI slowdown. I've always asked "What's in it for China? Why should China agree?" If the idea is "Make sure all AI globally is stuffed full of the kind of anti-everything bad, pro-everything good where 'bad' and 'good' are judged by "what would get the San Francisco Board of Supervisors seal of approval?" then is anyone really surprised the CCP is "Um, we're not so sure about that"?

What's in it for China? Why should China agree?

Well, they presumably do not want to be turned into paperclips.

We believe in pluralism. Regardless of if you're a fan of Scott Weiner or Connie Chan, you're welcome in our big tent.

Yeah. If Dario cared at all about AI safety the first thing he would do is step down, or at least stop making public appearances. His bad comms have done untold damage to the AI safety movement.

So what's up with him? Does he lack the wisdom to see the damage he is doing? Or is he power-seeking?

Main character syndrome. Even the founding of Anthropic itself was an error (maybe the fundamental error) from a safety point of view: having some differences of opinion around product safety decisions is not a reason to start a rival lab creating exactly the kind of competitive dynamics people have warned about for decades.

I genuinely think that for these people, the world does not exist outside of the Bay Area/Silicon Valley bubble. Oh sure, Washington because the politicians, maybe New York for money, but the 'real world' is only the one they personally inhabit and the people they interact with, the rest of us are just NPCs.

‘America B’

They could but how well equipped are they to judge Trump-whispering skills or redneckism?

Imagine the opposite. Would the rednecks know which blue-haired transgender could get them moving in the Portland slam poetry scene (which they need for some reason)? There would be all kinds of problems I think.

Firstly, the rednecks don't really want to be there, it's a purely instrumental goal. Secondly, it'd be hard for them to tell who's good and who's bad, what do they know about this? Thirdly, they don't live near them, so it's a bit of a pain to work with them just logistically. Fourthly, they'd struggle to understand the do's and don'ts, what questions they even need to be asking, how to formulate their approach.

Fifth it would be embarassing to their redneck friends working with xi/xir. I think this is a big issue with the labs, they're already competing like mad for talented programmers. Young, hyper-educated STEM workers lean anti-Trump.

In the 2024 election cycle, employees of Google’s parent company, Alphabet, have overwhelmingly donated to the Democratic ticket, with approximately 98% of their contributions going to Kamala Harris.

And sixth the whole project is unauthentic. The slam-poetry scene can tell that these guys are rednecks, not really part of the community. They'd be on guard the whole time for whatever trickery or shilling the rednecks are doing.

This is extra hard for the AI labs since Trump-whispering is an international geopolitical affair. Everyone wants to be that man's friend. Everyone is going to say 'oh I can get along well with Trump' whether it's true or not. You really need to have good connections to make things happen. Aschenbrenner's finance friends can do that but Anthropic can't, I think.

Trump must have a million people trying to jabber in his ear to do X, Y, Z.

In the 2024 election cycle, employees of Google’s parent company, Alphabet, have overwhelmingly donated to the Democratic ticket, with approximately 98% of their contributions going to Kamala Harris

How smart can they be if they backed the no-hoper? That's snarky, but it's all of a piece with the Carrick Flynn election effort. Two minutes reading about the candidates there had me going "The union candidate is gonna get it" and lo, so it came to pass. All the smart, well-meaning, hopeful people with the FTX (ahem) money were looking in the wrong direction - at what they thought was The Mostest Important Issue Ever (pandemic prevention, remember those days? today they'd probably run someone on AI doom) and not "what are the voters in this new district in this state likely to feel are important to them?"

That's the trouble. "We are really really smart and we think this is the most urgent thing so everybody should think it's the most urgent thing and we don't need to know the views of others who don't think it's the most urgent thing because they're not as smart as us".

To quote "The Man Who Was Thursday":

Then all Bull’s boiling good sense and optimism broke suddenly out of him.

“Oh, this is all raving nonsense!” he cried. “If you really think that ordinary people in ordinary houses are anarchists, you must be madder than an anarchist yourself. If we turned and fought these fellows, the whole town would fight for us.”

...“I think,” said Dr. Bull with precision, “that I am lying in bed at No. 217 Peabody Buildings, and that I shall soon wake up with a jump; or, if that’s not it, I think that I am sitting in a small cushioned cell in Hanwell, and that the doctor can’t make much of my case. But if you want to know what I don’t think, I’ll tell you. I don’t think what you think. I don’t think, and I never shall think, that the mass of ordinary men are a pack of dirty modern thinkers. No, sir, I’m a democrat, and I still don’t believe that Sunday could convert one average navvy or counter-jumper. No, I may be mad, but humanity isn’t.”

... Dr. Bull tossed his sword into the sea.

“There never was any Supreme Anarchist Council,” he said. “We were all a lot of silly policemen looking at each other. And all these nice people who have been peppering us with shot thought we were the dynamiters. I knew I couldn’t be wrong about the mob,” he said, beaming over the enormous multitude, which stretched away to the distance on both sides. “Vulgar people are never mad. I’m vulgar myself, and I know. I am now going on shore to stand a drink to everybody here.”

To get anywhere, AI issue people are going to have to deal with and convince ordinary vulgar people, and they won't do that by appeals that work for a particular set of 'modern thinkers'.

Everyone wants to be that man's friend. Everyone is going to say 'oh I can get along well with Trump' whether it's true or not.

I seldom want to respond to a post with peals of laughter but this has to come close. Pretty much every visible government in the West doesn't like Trump and makes it a point of pride about how they don't want to get along with him.

I meant more 'Qatar gave him a whole aircraft'... There are presumably intelligence agencies working on manipulating or compromising him. Hard to compete with that as a private company.

I agree, but I think that with genuine care these problems are addressable. If you actually, sincerely, care about AI safety more than you do about not getting on with rednecks, you can go and meet these people, you can spend time with them, you can ask them sincere questions because you really care about getting honest answers. Understanding other cultures is tough and comes with pitfalls for sure, but it's still something you can do with a little care.

Just get your senior guys to turn up to a bunch of slam-poetry recitals to start with, get a feel for it, get the lay of the land, work out who's important. Actually being open minded gets you 50% of the way there. After a few months you can at least get a feel for which PR consultants are bullshitters and which ones are authentic.

Fifth it would be embarassing to their redneck friends working with xi/xir. I think this is a big issue with the labs, they're already competing like mad for talented programmers. Young, hyper-educated STEM workers lean anti-Trump.

This is the main constraint, I think. The AI safety people only think their priority is 100% AI safety, it's actually about 60% AI safety; 40% not looking bad to their friends and getting redneck/trans taint on them.

You really need to have good connections to make things happen. Aschenbrenner's finance friends can do that but Anthropic can't, I think. Trump must have a million people trying to jabber in his ear to do X, Y, Z.

Yes, but how many of them potentially hold America's future in their hands? Trump and his administration clearly cares about AI to some degree, they've tweeted about it, served legal notices on it. The door is open, you just have to let them walk through it now and again, and be willing to say nice things when they do.

Trump is reTruthing Fox News clips framing AI safety as a global one world government Democrat control-seeking scheme...

How sure are we that he is wrong? The Amodeis haven't exactly been shy about their politics.

Suppose that AI risk is a hoax. Suppose that the alignment problem is easy.

Then Anthropic plows ahead, builds AGI, and takes over the world. Despite not falling for the Democrat “hoax”, we still end up with a Democrat one-world government.

The only winning move for Trump is to stop Anthropic from doing frontier AI research and/or nationalize the labs.

Yud had a bit on Twitter about that today:

I note, because I think that even now it still matters: On my history, EAs generally and Amodei specifically, indulged in politically polarizing "AI safety" in a leftist direction, because that got them short-term power; stymieing others' attempts to stay right-left neutral.

https://x.com/allTheYud/status/2099583457093734495

Oh God, it's finally happened: I agree with Big Yud.

A dark omen indeed

I am not sure at all that he is wrong. There are many reasonable reasons to believe this! I favour decentralization but there are structural reasons why this could be difficult.

I'm not keen on bringing forth alien RSI-sculpted entities at top speed, that doesn't seem very wise. This is a tricky bind.

I doubt the Left would be much better; it's deeply convinced that LLMs are just Gebru/Bender style racist stochastic parrots being pushed by NFT tech bros that have zero value of any sort, and existential safety is just a TESCREAL distraction from the real threats of climate change and Trump. On the whole, safety seems deeply out of fashion; if anything, people have updated against AI safety in the wake of the HF incident, which is just wild.

The only place existential safety oriented folks have a lot of sway is, ironically, OpenAI and Anthropic. sama is kind of a cypher, but safety is something he takes more seriously than you give him credit for, sandboxing incompetence aside. (A spicy take: he's objectively better for safety than Dario, because he's socially fluent enough not to alienate every potential ally or future collaborator.)

(A spicy take: he's objectively better for safety than Dario, because he's socially fluent enough not to alienate every potential ally or future collaborator.)

Hmm? Isn't it the case that basically everyone outside of OpenAI hates him with a passion...? At least Dario managed to keep hold of his fellow California progressives.

"California progressives" hate all tech CEOs, no matter how much the CEOs might genuflect before them.

Outside of Reddit, people think of Altman as just another typical CEO motivated by greed and power. That's fairly conventional and someone you can negotiate with. If there's some grand binational bargain to be made between the US and China, who does China want to be negotiating with, Altman or Amodei? Repeat the scenario for Trump, other corporations, etc.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats