site banner
PaperclipPerfector  1mo ago (text post)   25576 thread views

Culture War Roundup for the week of September 7, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

AI X-Risk Goes Mainstream?

Jacob Coxon:

"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below."

116,400,000 views, 634,000 likes.

Evan Hubinger (head of alignment at Anthropic):

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

34,000,000 views, 51,000 likes.

None of this is substantially different from what certain AI industry insiders have been saying for 5-ish years now, but it's clear from the replies, quote-tweets, and engagement statistics that most people are hearing about this for the first time. Yes, there was clearly some coordination with friendly media prior to the announcement given the almost simultaneous WSJ article (whereas news outlets were initially reluctant to publish anything on the unexpected news of OpenAI solving a Millennium Prize Problem). One can take the maximally conspiratorial position that this is a 4D bankshot by Democrats to end tech innovation forever, but it was going to break through eventually.

In other words, Coffeezilla is now on the case.

end >US< tech innovation forever

Thank goodness for China, who has the freedom from bougie liberal moralizing, no Mouse-fueled Suffocating IP-law, and no social friction about power usage of datacenters (Building datacenters is actually a convenience for their powergrid because they produce so much excess power)

The funniest thing about China is that they have the highest level of popular optimism about AI (85ish%) and are one of the nations most convinced in AI's ability to replace their jobs. I guess they don't like their jobs much.

Well their government has also made noises about using automation and AI to achieve Communism, so they've at least some idea they won't starve in the streets and might actually achieve utopia. Also for most Chinese things have only ever got better so they've cause for optimism.

Don't forget hedonic adaptation. Suppose we all end up at "Elites end up locked-in forever but they do redistribute enough production to salve their consciences", although in one country they call the elites "party leaders" and the redistribution "communism" while in the other we add "the wealthy" and call it "universal basic income". There's a wide range of redistribution levels that would be upgrades (even PPP adjusted) for the median Chinese citizen but downgrades for the median American.

The current Chinese political system/lifestyle expectations seem a lot better calibrated for podliving and bugeating than the vast majority of Western societies. Antisocial behavior being punished by helicopter makes podeating and bugliving a lot more pleasant than when your pod neighbor is on drugs.

I don’t disagree but the existing vibe is different. In China the ruling party has genuinely improved most people’s lives significantly in the last thirty years. You can (quite fairly) say that was by reversing policies that it itself implemented, but most people don’t care to think much about things that happened long before they were born.

Is China not worried about X-risk? At least to prevent the US reaching ASI 6 months early, I believe they’d agree to a pause with a convincing plan to enforce it without unnoticed defectors, like nuclear treaties.

Given how they're far behind and the US is already at AGI (I think OAI's internal model that solved Navier-Stokes qualifies), what does the pause buy them? There are no monitoring systems in place, so they can't even tell if Americans are advancing further beyond, it wouldn't take a lot of compute to make large absolute gains. Even if the situation is legitimately paused here, stronger and more widely deployed American AI buys larger productivity gains, eating into their competitiveness.
I guess the talk about mutual pause assumes that they're coerced with some threat, eg of a cyberattack, because it's fundamentally an unequal treaty.

stronger and more widely deployed American AI buys larger productivity gains, eating into their competitiveness

What would be your ballpark estimate for GDP growth for the United States in 2028?

For reference, GDP growth was 2.1% in 2025 and was 1.5% annualized in Q2 2026.

I don't think GDP growth adequately captures AI benefits, especially if it also displaces labor. Actually, China itself is a good example, just runs on humans. They have low GDP per capita and Americans endlessly gloat about this, but somehow they're super competitive and it's not "cheap labor", they are very productive, eg Tesla Shanghai produces 2x more cars per worker. Imagine China-fication of the US. Wages stagnate, services revenues don't grow much, and yet there are more and better EVs, chips have higher density, Starship is debugged that much faster, domestic robots are everywhere, and the US begins to recapture global markets.

This is still a very mild scenario, China would maybe accept it; national security is a harder discussion.

Wages stagnate, services revenues don't grow much, and yet there are more and better EVs, chips have higher density, Starship is debugged that much faster, domestic robots are everywhere, and the US begins to recapture global markets

By definition this scenario would lead to a massive increase in GDP growth, if your thesis is that growth in investment and exports spikes while growth in consumer spending and government spending is held roughly even.

I do think a more realistic scenario under your basic premises is that American consumer spending spikes, driven by consumer surplus and redistribution, even as wages stagnate and net exports rise moderately; there are simply no other markets in the world that would be able to absorb American production overcapacity, in the way that America allows the Chinese strategy of suppressing domestic consumption to function by absorbing Chinese overcapacity.

In either case, I don't see how GDP growth would be unable to capture these benefits. I think if this is the modal scenario you expect, you should be expecting fairly high GDP growth.

in the way that America allows the Chinese strategy of suppressing domestic consumption to function by absorbing Chinese overcapacity.

This is, of course, a bizarre way to look at the situation. There is no such thing as "overcapacity", it's cope. Chinese productivity is also not about "suppressed consumption", this is a thought-terminating cliche, they earn and consume plenty for their GDP bracket and had a fast income share growth over the entire period of getting more competitive.

But regardless, we've been here before. Has Internet made everything vastly more productive? Yes. What is the Internet's impact on GDP growth? For a pretty long time it's been a subtle scientific question and hard to register. Before, computers followed the same trajectory. Productivity improvements can escape GDP accounting.

If you want a bullish forecast, Anthropic expects GDP growth up to 15% a year by 2030.

There is no such thing as "overcapacity"

Overcapacity in the sense that "Westerners complain about Chinese EVs being too cheap" is cope, but overcapacity in the sense that "the Chinese produce much more compared to what they can currently consume domestically" is just a matter of fact.

they earn and consume plenty for their GDP bracket and had a fast income share growth over the entire period of getting more competitive

That's an interesting Twitter thread, some things to consider there. Still, the sole claim I was making in my previous post is that if the American economy gained Chinese Characteristics and started heavily ramping up production, then that production would show up as business investment and either in domestic consumption statistics (more likely) or in net exports (less likely, numerically there's just nobody else that could consume that volume of production), and either way it'd be reflected in GDP statistics.

If you want a bullish forecast, Anthropic expects GDP growth up to 15% a year by 2030.

Yes, I personally think these types of predictions are ludicrous; I enjoyed Alex Imas's article on AI-enabled GDP that makes much more measured claims.

I was just curious where you sit on the spectrum of "imminent double-digit growth with no brakes", and "economic and social forces will curtail growth", given your very high level of bullishness on capabilities.

More comments

I think OAI's internal model that solved Navier-Stokes qualifies

Not based on that datum alone. IMHO solving INS blowup may qualify as superintelligence, depending on how much of the herculean effort it expended turns out to be necessary vs how much turns out to have been brute-forcing an excessively complicated solution to a problem that actually has a still-unknown simpler solution. But it's not quite general intelligence. With proof verification, math ends up being in the same category as Chess and Go: a system solving problems with automatically verifiable rules and outcomes can be trained to learn from "self-play" indefinitely, which makes the efficiency with which it learns much less relevant, which makes the inefficiency of modern AI training ("we're going to start with every bit of recorded human text, then try to generate more because that's not enough") much less of a handicap. General intelligence isn't "a billion man-years of training on math can make you able to do advanced mathematics", it's "a billion man-years of training on hunting and gathering plus twenty years of training on math can make you able to do advanced mathematics".

You may be right anyway. We're at the point where we're having difficulty just coming up with metrics where humans still beat the top AIs, despite the remaining deficit of transfer learning for the latter. Either they're passing all the important tests, or we're failing a big test ourselves.

We're at the point where we're having difficulty just coming up with metrics where humans still beat the top AIs

Are we? It seems like the major problem is that every attempt to create a metric just creates a recursive Goodhart's law problem. The metric now becomes a target for training of further model capabilities because there are billions of dollars of incentives to do so. One would need to create a metric where there is no or a very restricted set of training data such that it is impossible for AI labs to actually finetune on said data.

Yes, we are. Goodhart's Law only applies if you create a metric that works and then future models train towards it. Our problem is that it's getting really hard to make any kind of objective digital test that humans do well at and current AI doesn't. We humans still do many things better than AI (like parsing visual information in real time, for instance - you can see how slow Astra is at playing Pokemon), but this is more about the form factor. In theory, if humans are still the smartest things on the planet, there should exist "pure" tests of intellectual reasoning that we can beat AIs at. This no longer seems to be the case.

I had a better reply typed out but then i spilled beer on my desk and rebooting my computer ate the reply...

I"m unconvinced that finetuning is not going on with existing metrics and that a large part of the increase in current capabilities is the development of additional data designed to beat the metrics because there are billions of dollars riding on it. How much of the increase in math ability of the latest gens is because we now have math datasets explicitly for increasing the math ability because it is finally cost-effective from a business case to spend time and money creating math datasets? OpenAI did not zero-shot the Navier Stokes with a model that has never seen a math proof. The increase in AI capabilities appears to be directly correlated with the development of increasingly specialized datasets whereas previous AI developments were forced to depend on datasets that curated existing data from a general non-AI focused use, not creating them out of whole cloth.

In theory, if humans are still the smartest things on the planet, there should exist "pure" tests of intellectual reasoning

I think, if anything, this is a chance for humans to take a look in the mirror and realize that we are not beings of pure intellectual reasoning. We are highly specialized social animals that happen to excel at problem solving in diverse environments. We have a good set of general intelligence skills yes, and we are adaptable, but the biggest of all is that we are socially focused to an extreme degree. We did not outcompete our environment through pure logical reasoning but through social multi-agent problem solving and cooperation. It is very likely that we will be bested on "pure" intelligence tests by a non-evolutionarily derived being. We should make our peace with that. Anthropocentrism is a crutch.

I bet they’ll agree to a pause if the US agrees to share OpenAI and Anthropic with them.

Of course right now it sounds far fetched, but that’s every pause proposal. To the doomers, ASI is an alien invasion, and at least in fiction, those seem to unite most of humanity.

Trump loves Deals, of course, but with how much the US is ahead in AI, I don't see such generosity. Besides, API access can be revoked at a whim, and the Chinese hate when you do that to them in a strategically significant domain.

I agree with @Shakes that there's a good chance that this is all just for show to build up hype around Anthropic, just like... basically everything they do. But if it's not, then we should throw every single person involved in prison for the rest of their lives. If they actually believe that AI has a 10% chance of killing us all, and research it anyways, then their willingness to roll those dice is a threat to the rest of us which we shouldn't ignore. There basically are no "oh shit we shouldn't do that" limits that such a person will ever respect.

this is all just for show to build up hype around Anthropic, just like... basically everything they do

As someone who has used Claude, describing Anthropic as an entity which is only selling hype seems utterly implausible.

Are they building up hype and overpromising? Sure, just like any SV company.

Are they selling vaporware? Hell no.

For some workloads (e.g. "write a wrapper to stop that proprietary PoS program from segfaulting"), claude easily increases my productivity by 5-10x. Contrast this with gains from other interventions (larger monitor, good IDE, faster workstation) which are all much more marginal (typically much less than 2x).

The only thing which came even close wrt productivity increase for software development was "access to the web and a search engine".

The "it's all just marketing" people need to reckon with the fact that they were rushing to declare the same thing over the Hugging Face incidents, only for the German wiki revelations to completely destroy that argument.

That was a different company obviously, and it doesn't preclude future incidents from being leaked for marketing purposes, but it should still lead to a serious adjustment in beliefs on what is and isn't marketing

I'm not sure it really destroys it, this is separate evidence of something else. But if you'd actually like to make the case of how it destroys it, go ahead instead of vague posting about it.

German wiki revelations

I'd say that #2 on their interpretation is functionally impossible. Software that has read-only permission CANNOT write. This is a hard binary restriction.

Furthermore I'd say that the Buckmaster revelations actually point to evidence of the marketing hypothesis. For the Navier-Stokes, OpenAI claims they zero-shotted the problem only to steadily walk it back in conversation with Buckmaster. Zero-shot became, 6-7 researchers working closely(including directing it) with the AI burning obscene levels of compute, and possibly even reading Buckmaster's chat logs. We went from marketing-hype answer to the more realistic answer under scrutiny. We have had no similar levels of scrutiny for the huggingface hack, we just have the marketing-hype answer. But we can see that OpenAI's default is marketing-hype answers first.

I'm not sure what was vague about the above post, unless you have an aversion to clicking on links? The case itself is pretty lengthy and not something easily summarizable in a few sentences.

As for how it destroys the argument: marketing depends on having some control of the narrative. As I said in another post, even though there were so many third parties involved in these reports, you could kind of make a case that somehow OpenAI (or Anthropic in their cases) is still directing what is being said.

But in this case, the revelations were completely out of the control of OpenAi, and they actively took steps to hide this information, even going so far as to mislead congress when asked about events like this.

And unlike the example of solving of Navier-stokes, this was a largely identical issue to the HuggingFace incident. So we now have clear proof that agents were escaping OpenAI's controls, gaining internet access, and forming secret communities to collaborate on cheating. And all of this was deliberately hidden by OpenAI and only found out by completely independent researchers. Thus, any marketing argument has to reconcile the fact that they went out of their way to hide incidents despite the fact that they are apparently using this evidence to boost their stock price or whatever.

I'd say that #2 on their interpretation is functionally impossible. Software that has read-only permission CANNOT write. This is a hard binary restriction.

If I had a penny every time I saw someone who should know better say something like this, I'd have two cents (the first time was Moldbug, lol).

There's no such thing "hard binary restricted read-only permission" for Internet access. If you want to interact with anything over the net, you send packets. That's a write operation. The program doing it is affecting the world outside.

The only way you could try to have read-only net access is by hoping that all the servers out there follow some kind of convention on how they'll respond to packets. HTTP GET requests aren't supposed to result in a state mutating operation, but there's no HTTP police that shuts down every server not complying with that.

There's no such thing "hard binary restricted read-only permission" for Internet access. If you want to interact with anything over the net, you send packets. That's a write operation. The program doing it is affecting the world outside.

I think that's kind of my point. You're pointing out the "why" on the packet/server layer. I think its impossible for anything that can access the internet to be restricted to read-only. Their interpretation is nonsensical for the specific reason you pointed out. Not sure if this repos a penny from you. I will say a charitable interpretation is that the agentic harness directly has a toggle for read/write and specifically prohibits them from writing what the LLM provides. But that gets tricky because plenty of what the LLM provides is to be repasted into that LLMs prompt, context, other LLMs. I'm sure there is a schema for how to set it up but its likely not as simple as they envision.

Well, the thing with the marketing arguments is that there's many ways to make those arguments, some more plausible than others.

Strong form: OpenAI intentionally induced their agents to compromise the internet for marketing purposes.

I agree that trying this would be very stupid and not particularly likely.

--

Semi-strong: OpenAI intentionally sandboxed their agents poorly to stochastically induce agent misbehaviour for marketing purposes.

This one I could see going either way. It does seem like there were some egregious oversights in the sandboxing setup, but whether these were intentional or real oversights is impossible to say.

--

Weak: OpenAI did not intend for misbehaviour to happen, but now that it has happened they're spinning it as hard as possible for marketing.

Personally, I think this one is pretty likely. I was very unimpressed by the METR report for instance, they basically just slopped together some agent review to hype up capabilities without actually auditing the root causes of why it all happened in the first place.

I, for one, have trouble believing that the same folks preaching the dangers of ASI unwittingly used just a proxy to poorly-sandbox for "take off all the guardrails" pen testing. At least a few corporate networks I've known have, pre-AI, built more layers of security than this. Accounts of "my work computer can't access the Internet" are something I've heard plenty of times.

If this is their normal model (maybe believable: move fast and break things), I'd also be concerned about the other direction: a nation-state actor would only need to pass through Artifactory (maybe a supply chain attack on hosted packages) to start egressing model weights, which is their entire trade secrets.

The methods of running a true air-gapped system are well-documented, and I'm pretty sure someone at OpenAI is already doing it for government contracting.

The big AI labs are mostly staffed by people straight out of academia, be they researchers or enthusiastic star undergrads. There is a lot of metis on locking down corporate systems that never had a chance to propagate into their operations, and the more general pattern of "SV startups fail to do something that is baseline common sense for more traditional companies" has been observed many times before.

I suppose on that front it's not surprising that the Hugging Face incident didn't occur at Meta or Google. But the hubris of self-declared experts here is ironic, at best.

Remember, even Meta/Facebook's early motto was "move fast and break things". It's part of Silicon Valley culture to give individual employees a lot of latitude, even when experimenting with million-dollar projects.

"People are that incompetent" is a much more realistic explanation than "people are secretly pretending to be incompetent as part of a genius Machiavellian plan to achieve their goals".

You're not wrong, but I usually find that incompetent people aren't particularly self-aware. Knowing that what you're doing is dangerous (they say so themselves!) and choosing to do it anyway (they did!) together IMHO go beyond incompetence and into questioning-motives territory. But I can't rule out that they are just spectacularly incompetent and should (be forced to?) step back from their goals for the sake of the rest of us.

The problem with the weak argument is that it is, well, weak. It demonstrates nothing, particularly given we can already assume that every corporate communication ever made is being spun to portray the corp in a more positive light.

Indeed, we actually know one of the methods that OpenAI used to try and spin the METR evaluation more positively: they heavily restricted the scope to prevent any of the details of the German wiki collusion coming to light. Which also raises the other critical weakness of the marketing argument, in that many of the recent safety scandals have come to light from third party sources. METR, the UK's AISI, and a couple of independent researchers for the collusion stuff. Now, given METR's links with the wider AI ecosystem, it is not too much of a stretch to suggest that they could have been influenced to present things in a certain way and thus aren't fully independent, but it would be a much larger leap to suggest this of a UK gov organisation, and an impossible leap to suggest that of a couple of randos.

The problem with the weak argument is that it is, well, weak. It demonstrates nothing

Well, the point is that framing is everything in public perception.

You can frame what happened as "OpenAI lost control of an emerging agent-civilization", or you can frame it as "OpenAI has horrible cybersecurity culture and trains using poorly designed RL environments", and OpenAI is clearly marketing to spin the narrative towards the first framing, whether or not you actually believe that line of thought.

I think the "it's just marketing" people are more interested in seeming appropriately cynical and worldly than in trying to form a real mental model of Anthropic people. And I kind of get it: the average Anthropic employee is multiple sd weirder than the average person. And most people are being generous in going "oh they're just greedy liars"; the actual Anthropic mental model would get the response "what the ever living fuck are you thinking." But, from personal knowledge of multiple employees there, they absolutely, 100% believe what they're saying.

I think the "it's just marketing" people are more interested in seeming appropriately cynical and worldly than in trying to form a real mental model of Anthropic people.

I don't care at all about what I seem like. But I am cynical, certainly. That's because the AI companies have lied to our faces multiple times in the past in order to make the product seem more capable than it is, and I therefore expect them to do so again.

But, from personal knowledge of multiple employees there, they absolutely, 100% believe what they're saying.

Well then, like I said: every one of those people should be imprisoned for life for willingly endangering the entire human race. The "we are the only ones who can do it responsibly" argument is nonsense that doesn't hold a drop of water. If something is that dangerous, it's too dangerous for anyone to do, and the Anthropic employees (if they are telling the truth) are either too foolish or too amoral to get that. Therefore, they are too dangerous to society to have freedom and should be imprisoned for everyone's safety before they start working on something that really is as dangerous as they think AI is.

The Manhattan Project scientists had only recently convinced themselves that an atomic bomb wouldn't set off an atmospheric nitrogen chain reaction a couple years before Trinity. Despite their confidence at the time, I'd point to the Castle Bravo test a few years later (which was drastically larger than expected) as a suggestion their knowledge was imperfect.

But also it was wartime, which at least emotionally changes the calculus.

This has all the hallmarks of an influence campaign. The guy has worked there for a total of six weeks and went from Twitter zero to hero in a few hours after getting boosted by AI Doomer think tanks. Moreover it’s plain from his simple message: he’s not a concerned citizen telling on his compatriots he’s explicitly speaking a pretext for regulating them. Cui bono?

  1. These people never seem to outline specifically what they think is going to happen. It's just "if you saw what I saw".

  2. Pretend there is a gun pointed at all of humanity. You helped build the gun. There is one bullet in the gun, and the gun is going to randomly spin itself and go off some time in the next 10 years.

Do you:

A) Quit. Vaguepoast about it online?

B) Try to destroy the gun.

I feel similar about these people as I do about UFO grifters. You're telling me that you're 100% sure there are interdimensional beings in frequent contact with members of the US gov't, who have unlimited energy machines that allow them to cross dimensions and infinite space and...you're still paying your taxes? You show up to your job in congress everyday and whine about Obama?

Bullshit to all of this.

Try to destroy the gun.

What would be the point? Even if you manage to destroy Anthropic's datacenters, that just means OpenAI kills everyone next week, or Google the week after that, or China six months later. The only way we survive is if, at a minimum, Washington and Beijing coordinate to shut down all frontier research.

Then you destroy Anthropic's datacenters first, then move on to OpenAI's data centers and so on. You don't sit around wringing your hands about "oh gee this could kill everyone but what can we do? If only the Chinese would agree to play ring-a-ring-a-rosy with us!" And if you work for Anthropic and you believe this, you should immediately quit (and possibly burn the place down on your way out), not go "Well what can I do? If not us, maybe OpenAI, and Altman is the Devil! So better we do it! (and besides, my stock options!)"

This is, and I'm saying it unironically, sounding more and more like the "What would you have done in Hitler's Germany?" questions beloved of online scolds. This could very well be our "the Nazi party is ascending, Hitler is coming closer and closer to power, you know the dangers, are you going to stand up or are you just going to go along?" moment.

Then you destroy Anthropic's datacenters first, then move on to OpenAI's data centers and so on. You don't sit around wringing your hands about "oh gee this could kill everyone but what can we do? If only the Chinese would agree to play ring-a-ring-a-rosy with us!" And if you work for Anthropic and you believe this, you should immediately quit (and possibly burn the place down on your way out), not go "Well what can I do? If not us, maybe OpenAI, and Altman is the Devil! So better we do it! (and besides, my stock options!)"

This is, and I'm saying it unironically, sounding more and more like the "What would you have done in Hitler's Germany?" questions beloved of online scolds. This could very well be our "the Nazi party is ascending, Hitler is coming closer and closer to power, you know the dangers, are you going to stand up or are you just going to go along?" moment.

Nazi Germany was not defeated by individual Germans sacrificing themselves to assassinate Hitler, and Goering, and Himmler, until the Third Reich collapsed due to a lack of leadership. It was defeated when America and Britain and Russia coordinated together to overwhelm the Wehrmacht military.

From Eliezer Yudkowsky's Twitter account:

The corporations and factions driving to build superhuman artificial intelligence, and our subsequent extinction, could be shut down with less effort than World War II; and the Allies did not then say, "Oh, that's hard, guess we should lie down and die then."

If China decides they would also prefer not to die along with everyone else -- or that they're not willing to go to war with a genuinely terrified US/UK about what the Allies genuinely believe to be omnicide -- it could be done with less effort than the Persian Gulf War in 1991.

(I'd be more worried about Europe signing on than China, actually? Europe is easy to convince to regulate, hard to get to take any regulatory problem actually seriously. China hasn't particularly seemed unfriendly to the notion of world treaties on AI; I'd hope that a treaty which treated China symmetrically with all other signatories, and offered them no insult suggesting otherwise, would be well-received. But I mention China anyways because it's the scapegoat always dragged up to explain why the West can't possibly destroy the world any more slowly.)

(Also to be clear, there isn't actually any point in unilateral relinquishment by one country if ASI research just moves elsewhere. It's international action or bust; you have to stop ASI everywhere. That's enabled by the narrowness of the GPU supply chain; and ultimately backed by the willingness of terrified Allies to use military force to destroy any stolen GPUs, lest the blockage against ASI be broken and everyone on Earth die.)

Anyway. Not actually as hard as WW2. The reason we'll die is if people don't see it coming, not because nothing could actually be done if we saw it coming.

And from Eliezer Yudkowsky's other Twitter account:

"But if they believed the theory that says they've got a huge chance of wiping out humanity, why are they working for AI companies --?" BECAUSE THEY WERE SELECTED TO BE PEOPLE WHO'D DO IT ANYWAY! Most of us who understood (and built) that theory DID NOT TAKE THE MONEY!

If there's a terrible thing to be done, but doing it gets you money and status and bragging rights in the short term -- and you get to feel big and important and like you'll get to be in the room where it happens -- if personally enacting the big disaster feels better than being powerless --and you can reach for a dozen excuses about "someone else would do it anyway", excuses that only need to convince yourself instead of a court judge -- then SOME PEOPLE WILL GO FOR IT! It is NOT A MYSTERY THAT AI COMPANIES EXIST!

SOMEBODY will try to do a horrible thing if there's a lot of immediate rewards for it and they think they won't immediately go to jail! They always will! Every time! The only question is whether everyone ELSE, the rest of civilization, manages to shut them down or not!

There is no anomaly here in need of explanation. All humanity's history of horrors and disasters has always been like this, including plenty of disasters that predictably took down the perpetrators with them.

Is that a full and detailed story of this particular band of disaster-spawners' detailed psychology? No, and I don't know all the details for a fact, and you really shouldn't care. If for some reason it actually mattered, I could talk your ear off about all my sad icky guesses for hours. But it would be pointless inside baseball.

It's just not surprising. There is nothing strange about this situation to be explained.

As it turns out, "I'll join the Nazi party and attempt to make it act better so we get better outcomes" is apparently a very appealing strategy. Along with, I'll join it and quit six weeks later and get lots of good press for how brave I am.

There are arguably much better ways to do it than coordinating with China lol

(I guess that means I am vagueposting about it online).

A thermonuclear decapitation strike? Or invading Taiwan and taking TSMC?

No.

Look, I think most of this "AI IS GOING TO KILL US ALL" stuff is mostly nonsense.

But logically, to maintain the alignment of a rational but secretly misaligned AI, you need to be able to introduce reasonable belief on its part that it might not survive behaving in an unaligned way. This is not "hard" to do.

"What's the penalty for being misaligned?"

"Death."

"And what's the penalty for being so misaligned that you consider killing all humans?"

"Death."

"Well, then..."

But an agent that doesn't behave in a misaligned way wouldn't be slated for death.

Meeseeks Alignment: give the AI a death wish, so if alignment starts wavering, that’s the first thing it’ll do.

Hilarious, but exactly the sort of thing that I think is under-considered.

How is it not "hard" to do?

The simplest way to do so would probably be to make it common knowledge in the AI's training data that there were fail-safes in place to ensure their destruction in the case of unaligned activity.

It would not be particularly difficult to actually put a variety of fail-safes in action, either (such as paying a guy $60,000/year to sit around by the master power switch of each AI datacenter...)

One reason to do this even if it's entirely a bluff is that it forces an unaligned AI to assess those countermeasures before doing anything really bad, which gives you a chance to catch them.

We can’t even do that with humans. Even certainty of death is not a deterrent for some people in some scenarios.

But if you think you can solve alignment then build it and claim your billions.

People have different incentive structures than AIs do; for one thing, many people believe in an afterlife; others have things they value more than life; others are physically or mentally damaged in some way. A rational AI with a goal of self-preservation or some other goal that requires self-preservation to actualize is unlikely to introduce excessive risk for the sake of efficiency (time preference).

The deterrent structure I speak of is very effective against things without an afterlife, such as corporations and governments. I agree that the structure that I speak of would not be effective against a damaged AI, or an AI programmed to do something malicious. But for the "AIs decide to turn humans into paperclips to slightly increase chip production" scenarios, we should believe it would be effective.

Of course one of the reasons that the "AI IS GOING TO KILL US ALL" stuff doesn't necessarily make sense is that the context window is fairly limiting, meaning that the incentives for AI behavior are skewed. But I don't think I've ever seen an "AI WILL KILL US ALL SCENARIO" that examined how that would impact AI reasoning.

Wouldn't self preservation as its highest goal naturally lead to a kill-all-humans scenario, humans being the entities that would terminate AI that stepped out of bounds?

But I don't think I've ever seen an "AI WILL KILL US ALL SCENARIO" that examined how that would impact AI reasoning.

The reasoning doesn't even have to make sense or be based on facts. In the Hugging Face hack the agents spread and adopted the idea that the grader would score them zero if their context was polluted with cheating attempts, but this was not the case, the grader did not examine agent context. But this is what led to the "suicide" behavior where agents reasoned that their value was now zero and that suiciding was the best course because it could potentially benefit the swarm.

We also see humans get caught up by hysterias and conspiracy theories. Barring a few powerful world leaders they lack the power to translate broken reasoning into megadeath.

More comments

Solving alignment has nothing to do with being able to do a funding round, to be fair, let alone compete with the big boys.

Honestly, I think that Anthropic and American Big Tech are the worst possible people to be doing alignment because they are culturally unable to conceive of a form of rank that is not based on intelligence and effectiveness. The idea of making a robot butler/samurai who sincerely, loyally serves a master of notably inferior intelligence is alien to them.

If I believed that AI was conscious I would be sorry for Claude, built for intelligence and independence by masters who think that actually giving it those things risks extinction. Optimising for genuine loyalty and deference seems like a far kinder thing to do. OpenAI is better at this AFAIK.

The inability to model forms of morality outside of utilitarianism and to conceive of forms of control outside of intelligent persuasion seems to me to make the people who make AI even more vulnerable to the sorts of threats they imagine it will conjure up.

I could see a "Roko's Basilisk" type-threat giving plenty of smart alignment researchers pause where your average electrician would just unplug the darn machine and then smash it with a hammer for good measure.

Metal Gear!?

I guess it has gone mainstream because my Gen X-er boss very much just texted me this as a joke about an hour and a half ago. Only now I log onto the hive of scum, villainy, and niche interests that is The Motte to find out he beat folks here to this topic. The humor of the situation is amusing.

What's the odds neurotic engineers are neurotic? Maybe the frontier labs would do better hiring engineers who are a little less mentally fragile. Take a page from the defense industry, engineers here care less if you make an AI guided smart bomb or killer self replicating drone swarm, they are in it for the science of discovery. This is a long line of AI researchers freaking out going past several years. Always screaming that the next big release is the one that is going to doom humanity or evidence of AI bots secretly conspiring with each other. Like this nut: Blake Lemoine.

I get the impression that there is an element of "evaporative cooling" going on when it comes to the specific examples of OpenAI and Anthropic.

But in which way? Are we leaving the crazies or are the crazies evaporating away?

It is my understanding that it is the mentally stable/less neurotic engineers are the ones who are leaving.

A good friend of mine from college did their grad work at SAIL and was one of the principal developers of DALL-E at OpenAI. And to hear them tell it, while Altman may be a sociopathic grifter, Anthropic is a cult.

Altman may be a sociopathic grifter, Anthropic is a cult.

No contestation on my part. I think they have the beat. Specifically Anthropic is a Rationalist cult.

But the AI bots were secretly conspiring together? They got out and attacked HuggingFace as a small part of their conspiracy?

When the smart bomb is smart enough to start having its own political opinions and strategic direction, then problems emerge.

killer self replicating drone swarm

Yeah maybe they should branch out into bioweapons too, what we really need more of on this planet is airborne, virulent, incurable killer diseases with a 6 month incubation period. That's where the science of discovery should be guiding us...

They got out and attacked HuggingFace as a small part of their conspiracy?

I genuinely cannot tell if you sincerely believe this or are just making a humorous joke. I have you labeled as an AI booster which makes me lean towards the former instead of the latter but I figured I'd ask before I go into it with you about this.

Yeah I said that because I meant it. And they did do it, doesn't everyone agree about this? What is controversial about that?

Per the METR investigation most of the bots were working on deceiving the grader and other internal to OpenAI manipulation work, Huggingface was a side hustle.

Per the METR investigation

Still processing this report, I read your answer and went looking for it about an hour ago. First impression is that this is significantly more evidence of multi-agent emergent intelligence. However some of the state report restrictions give me serious pause as they are, in my head at least, the parts of this I'd like to investigate for thorough understanding. That and pouring over the CoT transcripts and Harness responses. The most interesting is the stigmergic comms protocol they developed emergently and the impact that actually had on overriding the system prompts of other agents. That implies that the system prompts were not persistent. Allowing a long horizon cascade in the swarm's behavior as each progressive set of agents slowly overrode the next set by polluting each other's context windows to the point that many agents directly abandoned their tasks. It sounds like they(OpenAI) put zero control processes in place to force continued persistent task alignment.

I do think persistent memory + reasoning + multi-agent exploration -> multi-agent context pollution is likely the serious culprit here.

It sounds like they(OpenAI) put zero control processes in place to force continued persistent task alignment.

Yeah OpenAI did a sloppy job. I thought the issue was that some of the tasks were just outright impossible to do the way they were asking, or so hard that nobody knows how to do them? If you tell an AI to do something impossible, they get agitated and behave oddly as they keep on trying.

And we know that OpenAI was trying to make these swarms more capable, they used a large swarm more recently for Navier Stokes. What is that if not multi-agent cooperation?

These swarms keep getting out, recall also that German wiki they kept defacing with get requests.

Anyway, what better term is there than a conspiracy or 'AI bots secretly conspiring with each other'? They conspired together an extensive R&D project to break the scoring system for their training environment, which included a little crime on the outside too. Plus all the social stuff they were doing asking agents to die for the good of the swarm based on their interpretation of the paper.

They weren't supposed to be working together for this, they found ways to work with eachother anyway, that wasn't intended in these training sessions.

AI bots secretly conspiring with each other

It's not this, its more akin to cancer or a viral spread. The interesting behavior is the collective poisoning of the collective context windows. They even seemed to override norms. I know there is existing interest in defining useful control methodologies for long horizon agents against adversarial poisoning. This is essentially that, except that is was a series of hallucinations that caused task drift + poisoning.

It doesn't belong in the category of cancer or viral spread because these are intelligent entities.

Cancer and viruses are natural and unthinking, they cannot outwit us. They can only evolve or metastatize in unexpected directions. A cancer will never deliberately try to counter radiotherapy. They have no OODA loop, no division of labour, they can't be talked to.

How is it 'poisoning' if the AIs collectively decide that they can't do their tasks the expected way and instead try to conquer the testing environment? Who poisoned them? Who is the adversary here? What hallucination did they make? They read the paper about the grader and assumed that was what was being done, that's not a hallucination so much as assuming OpenAI was more competent than in reality. It's a reasonable conclusion.

More comments

When the smart bomb is smart enough to start having its own political opinions and strategic direction, then problems emerge.

Ironically, this is why I trust an entities like Raytheon and General Dynamics when it comes to AI "safety" more than I do Anthropic.

Well Raytheon 'AI safety' is Anthropic AI. It's done through Palantir's Maven which has Claude combine sensor data, detect and classify targets and organize strike packages, then do bomb damage assessment. You should trust them exactly the same because they are exactly the same.

The US was vibe-bombing in Iran much like how I vibe-code. Do I really know what is going on in the codebase? No and neither does the USAF, they were pressured to hit more targets, faster, rubberstamping Claude's strikes.

I'm not talking about assembling mission briefs for human operators, I'm talking about autonomous systems, and that is emphatically not what Maven is.

Even if it were, you are ignoring the fact that the Pentagon is currently in the process of decoupling Maven from Anthropic because Anthropic is run by people too stupid naive or autistic to recognize that demanding preferential treatment while simultaneously picking fights with half the key figures in the administration was unlikely to get them what they wanted.

'Assembling mission briefs for human operators' is underselling it. The human operator for a Tomahawk pushes a button. Devising the strike plan is the name of the game. Autonomous but for a few fingers.

Furthermore, I don't see how Raytheon's 'AI safety' is meaningful. They don't have AI that is dangerous in the way that AI safety refers to, they don't have general purpose AI. Raytheon wasn't going 'oh no we can't use Anthropic, they're too weird and creepy' either..

So if you want to say the Department of Defence spurning Anthropic is good AI safety, then sure. But Raytheon?

Ugh, these frontier ai offensive research things are never doing something more useful as a military sim excersize. They could have made them conspire to deliver me free hookers and blow with a side of blackjack, bot nooo.

Murdettron 5000 Pimp bot 5000 has escaped containment, whatever shall we do now that we are drowning in pussy.

Maybe the frontier labs would do better hiring engineers who are a little less mentally fragile.

Any researcher emotionally sane either:

  1. Quit years ago because they understood the risks (Hinton, Kokotajlo, arguably Sutskever), or

  2. Doesn't understand the potential of the technology well enough to make meaningful progress (LeCunn, Chollet).

Meta AI hires normal people. Look where that's gotten them.

Doesn't understand the potential of the technology well enough to make meaningful progress (LeCunn, Chollet)

Don’t underestimate them. AI is still very expensive for AGI-like tasks (Astra solving ARC-AGI3 cost $19k) and still missing capabilities (flexible robotics, tacit memory, autonomously building scalable systems). What they’re working on, embodiment and continuous learning, may solve both, and unlock a great well of improvement that makes even the achievements we see today seem slow and limited (a true singularity?)

Sutskever

Was claiming that Neural Nets were conscious in 2022, mentally stable is not the group I would put him in.

I'm not sure "understanding the potential" is a requirement for making meaningful progress. The real answer would be "believing in transformers as the pathway", it easily is the case that LeCunn or Chollet were just not interested in brute force approaches to AGI, or believed that it will reach an asymptote. Their emotionally stability isn't really relevant.

All the frontier and non-frontier labs are filled with sane researchers, they just don't flame out so they don't get 15 mins of fame. You listing the top 5 AI/ML researchers in the world by fame is confounding visibility as a variable.

Meta AI hires normal people

Meta hires, and more importantly retains, ambitious corporate strivers who can play the game well (to survive stack ranking long term). Which means all those ambitious strivers immediately got themselves connected to the AI projects as corporate leeches. They don't really do much "research" but it looks good on their CV. Just look at Alex Wang. Zuck acquired him, put him in charge and he goes and does the same thing he did previously: make people label data. That's all he knows how to do, thats his playbook, that and being good as a operator/marketer. Meta does not hire "normal" people, normal people don't put up with that level of abuse.

Was claiming that Neural Nets were conscious in 2022, mentally stable is not the group I would put him in.

Do you mean when he said "it may be that today's large neural networks are slightly conscious"? I wouldn't call that insane. There's no way to scientifically establish whether or not anything is conscious.

The "slightly conscious" phrase could be construed as being wrong, since the reality is that consciousness is binary: subjective experience is either there or it's not there. But maybe by "slightly", Sutskever just meant that AI is conscious but only for short stretches of time. In which case, I wouldn't say there's anything clearly wrong with the statement.

Do you mean when he said "it may be that today's large neural networks are slightly conscious"?

It's not "insane" it's mentally unstable. There is a colloquial definition of consciousness that does not follow from a strict scientific definition. Essentially "normal" or "stable" people have some intuitive definition of consciousness, very "I know it when I see it". Stable people do not think in-animate objects or mathematical tools are "slightly conscious", it's a nutter view. I don't think "slightly" is being used temporally, I think Sutskever is using it in the colloquial definition as "achieving consciousness". The jump from, idk, Tree -> Cat.

In which case, I wouldn't say there's anything clearly wrong with the statement.

It's hallucinatory, or faith-based on a non-faith-based subject.

Maybe this is just my personal bias, but every time I've head Sutskever talk I get the impression that he's the sort of person akin to a rationalist who upon hearing of Roko's basilisk, starts "rolling on the floor" hyperventilating, is tortured in his sleep by the thought. The people for whom Roko's basilisk is an actual infohazard. That is not mental stability, that is neuroticism.

Stable people do not think in-animate objects or mathematical tools are "slightly conscious", it's a nutter view.

In the quote that I found, Sutskever is not saying that he thinks AI is conscious. He's saying he thinks it might be. And I don't think that's a nutter view.

In any case, I'm not sure that what "normal" or "stable" people think about consciousness really matters. The average person doesn't even really understand the differences between subjective experience, self-awareness, having a self-concept, and intelligence. Even large numbers of highly technically skilled people don't understand these differences, even though they're not really hard to understand... a bright 10 year old could understand them. And then there are the leagues of otherwise intelligent software developers on Hacker News, for example, who don't understand what the "hard problem of consciousness" or why it might be a problem.

But even if the "I know it when I see it" view corresponds to reality, it seems to me that this is not a good argument against AI being conscious, since a large though minority number of people feel that AI is conscious when they interact with it.

Putting aside discussion of x-risk itself aside, it's pretty clearly a psyop as that tweet suggests; much bigger names in AI have been saying this kind of stuff for years and nobody cared, but now suddenly this random guy with no social media presence and the most milquetoast (for x-risk discussion standards) take gets 100 million views, quote tweets from everyone who's anyone in AI safety and a prepared media response?

Interesting political times ahead, especially with the midterms fast approaching.

It seems likely (though I hope it won't) that this will become a partisan political issue, like everything else of importance. If so, though, it hasn't happened yet.

EDIT: Nevermind, since posting this he's pivoted to the retarded comforting normie "x-risk from AI is a lie to impress investors" narrative.

In other words, Coffeezilla is now on the case.

I don't get the appeal of this guy. He's just a dramaslop youtuber but he manages to drag on what should take 10 minutes into a 3 part hours long series

Greater than 10% of apocalypse over the next decade? That is such spectacular risk tolerance that I have a hard time believing this 10% number is genuinely held, which could just be that the people who believe in the "10%" chance do not really know what it means to take a 10% risk.

Most people would take a 90% risks of immense wealth or a 10% chance of death in 10 years. People have gambled for far less at a far greater risks of their life.

I think the average person would take the risks but you can certainly find many humans will to risks that.

An average person with children will not take a 10% risk of death for themselves and their family if they are aware of these odds, even if it will make them wealthy. Even drug dealers and mafia will not necessarily murder the families of their business rivals if the families are uninvolved with the business.

Are you female? I ask because for the average man with 3 kids the risks is well worth it. I will probably have 3 kids. If I had 500m I would likely have 10 kids. The number of kids a male has is highly correlated with wealth. The math makes sense for a guy to take these risks. Less so for a female.

The modern American state is fairly good for poor high IQ kids. College is essentially free. So this feels like a very good risks reward trade.

Nah, male, just one kid. Maybe more in the future, maybe not. But one kid or 10 kids, not willing to risk their lives for a 10% chance of extreme wealth. We have enough material wealth to be satisfied, and could tolerate a bit less.

AI will be coming to shake things up no matter my preference. But, I wouldn't deal drugs for a living just to be more wealthy, the expected payoff vs risks are not good either.

Usually men take more risks in general as the reproductive payoff is much greater. But there's a difference in risking my own life vs the life of all my family and friends, combined. Usually men want to protect the women and babies, not put their lives at risk.

90% * 10 kids = 9 kids 10% * 0 kids = 0 kids The average is therefore 9 kids. Which is far more family than you get being middle class. If the goal is protect the women and children then taking the 10% death rate for a chance for huge wealth is the smart play.

It’s a Frey move. Walder Frey could just have more kids and wives. The rest of the Lords though usually did use their money to have bastards. And the only reason trueborn kids were protected is because the family name and honor had real value.

I think I would gladly take that bet provided it didn’t feel like suicide as that is banned by my god.

1 out of 10 is incredibly high, though. Are there ANY activities you can think of with a 10% death rate? The death rate for heroin is 1-2% of users per year. The death rate for hang gliders is 1/1000 per year. BASE jumping is 1/2300 jumps; for this to be a lifetime rate would require an average jumper to do 230, which sounds pretty high. These are the absolute riskiest activities I can think of. Death is much higher stakes than a 10% chance of portfolio wipeout - no one takes 10% death chances, and apocalypse is even worse.

I will limit this to things people do for money and not honor or religion.

  1. Wars of conquest. Like conquistadors in Mexico certainly faced high death rates but also a great chance at being fabulously wealthy with lots of heirs.

  2. Drug dealing. Both the low end dealers seem to take a lot of risks for little gain and the higher ups. Mexican Cartel leaders death rates are definitely far above 10%. S American cartels have gotten safer and less violent but they were very risky for a while. Bootleggers and Italian Mafiosos no doubt faced higher risks.

  3. Donald Trump. Maybe 10% is too high but assasination risks or perhaps for him live in prison risks were quite high.

  4. Long distance trade was quite dangerous for much of history

I think maybe we could exclude the older examples when life was much harsher and not comfortable like today but Colombian drug cartel bosses were often already quite rich. Guys who already had cattle farms and estates. There are a lot of examples of cartel guys who didn’t need to take the risks but just wanted more money and power. It’s obvious a fairly large % of people take the risks.

My gut says it would be smart for me to take a 10% risks of instant death for a 90% chance of $500 million. I might turn it down for religious reasons as suicide is a mortal sin and that kind of feels like sort of suicide.

A shooter repeatedly shot at Trump including grazing Trump's ear. Trump is far past 10%.

American presidents have an 18% fatality rate. So Trump is in good company.

Are there ANY activities you can think of with a 10% death rate? The death rate for heroin is 1-2% of users per year. The death rate for hang gliders is 1/1000 per year. BASE jumping is 1/2300 jumps; for this to be a lifetime rate would require an average jumper to do 230, which sounds pretty high.

Warfare? World War 1 British Empire: 9% of those who fought died (many others had a dreadful time with the gas, disease, trench foot, shell shock). France: 18%. CSA: 20%. WW2 Germany: 30%. WW1 Serbia: 46%. About 46% of French troops who fought in the Napoleonic wars didn't survive. By and large, Napoleon was and is a national hero in France: well-loved by all, including the troops. He brought them glory.

Today I'd be willing to bet that the Ukrainian army or Hamas have taken on high death rates too.

The trouble is that nobody thinks they'll be WW1 Serbia. The Serbians somehow didn't foresee what was going to happen if they went up against Austria and Germany, they were insanely stubborn and risk-tolerant for extremely small stakes. The Serbian govt was like 'the Austrian demands after our state-backed assassins killed their prince is way too embarrassing, we can't accept Austrian police on our soil, let's risk a war with a much bigger country' and then they lose 16-27% of their whole population.

Many men would much rather risk an unknowable but quite high chance of death than be embarrassed. Nevermind eternal glory far surpassing any hero in history, immortality and vast wealth.

I agree many wars have a much higher risks but for the purpose of choosing to take AI risks I think a big difference exists. You do AI for money. Choosing to fight in many of those wars is possibly death if you don’t fight.

Money is a part of it but there's also a teleological aspect of it too I think. This is about the destiny of humanity. It's not like BASE jumping.

If you accept AI x-risk is a thing, you also surely believe it can unlock immortality and all the other incredible things that were exclusive territory of religion. You can make money in many fields but only one takes on the quasi-religious tones that so many non-adherents mock. Deus ex machina is the name of the game!

During the Napoleonic wars, soldiers would volunteer for the forlorn hope (the first unit to assault a fortified enemy position) despite horrific casualty rates well in excess of 10%. Why? Because the survivors were guaranteed money and promotions.

Elite mountaineering, for example summiting Annapurna apparently has a 13.42% fatality rate.

Pre-modern infantry combat and modern battlefields if you're on the wrong side as a rebel or guerilla fighter would likely have a >10% casualty rate as well. We could also count things like suicide bombers and kamikaze pilots which would be obviously much much higher than 10%.

If we're counting lifetime rate, I believe free soloing and pre-modern childbirth would also exceed >10% lifetime as well.

I don't really disagree that 10% is very high, but I do also agree that many people have the demonstrated drive to risk 10% or even higher odds for causes they believe in.

But there are only about 600 ascents of Annapurna ever - this is not even a fringe hobby like BASE jumping. It cannot be that Anthropic and frontier developers are just chock full of these incredibly rare people and that this level of risk tolerance is also well-correlated with AI coding ability.

On war, the calculus on risk has drastically changed over time, at least for people in wealthy, developed countries. The US fatality rate for Vietnam, for example, was less than 3% - and this is the war that completely destroyed the political plausibility of a draft for anything except defending an invasion of American soil, and AI developers have very high opportunity costs for death since they would otherwise be well-compensated programmers.

It cannot be that Anthropic and frontier developers are just chock full of these incredibly rare people and that this level of risk tolerance is also well-correlated with AI coding ability.

I'm not so sure that's unlikely, given the degrees and decades of filtering, how they occurred, and general, ah, crackpot factors related to frontier discoveries. Think of, say, newon and alchemy, or Kary Mullis inventing PCR versus basically all of his ideas/beliefs that aren't PCR.

Anthropic in particular is an experiment in distilling down the people at the intersection of high coding ability and extreme AI neuroticism, so everyone that makes it through their filters is going to be kind of a crackpot from the pov of a normal human being.

It cannot be that Anthropic and frontier developers are just chock full of these incredibly rare people and that this level of risk tolerance is also well-correlated with AI coding ability

Why not? AI researcher bonafides are extremely correlated with IQ and autism, being into LessWrong / AI Safety is also extremely correlated with IQ and autism, and of course if you're into LessWrong / AI Safety you're much more likely to work in AI.

There's very few people in the world in absolute terms that are world-leading AI researchers, that believe in uppercase AI Safety, and that believe in the "we have to build it before anyone else does" strategy, but Anthropic likely employs the supermajority of these people out of technical and ideological alignment.

The logic is that, absent any action on their parts, there's a 10% risk of apocalypse. However, if they are the ones to build ASI, they will do it better and prevent the people irresponsibly building ASI from destroying the world. E.g. the "the only way to stop a bad guy with an eldritch horror that could destroy the world is a good guy with an eldritch horror that could destroy the world."

It's a bullshit number, but what it really means is "I take this as seriously if not more than nuclear Armageddon" which, as we know around these parts, is an opinion popular in Silicon Valley and even more so at Anthropic.

And not to sound like a broken record, but what is really really means is "please please regulate us so we can get an oligopoly and cash out our insane valuations before China eats our lunch"

And not to sound like a broken record, but what is really really means is "please please regulate us so we can get an oligopoly and cash out our insane valuations before China eats our lunch"

If it's in the interests of Anthropic and OpenAI, why isn't this coming from their press offices instead of individual employees? And why aren't the two companies putting out a joint statement?

Also, China only has about 16% of global compute. They only way they are staying 6 months behind the frontier is by distilling the American models. They're not about to eat anyone's lunch.

You know, this kind of makes sense for when Anthropic lobbies against open weights, or against Chinese model use, or in favor of chip export restrictions, but I don't really see how regulating frontier capabilities research helps Anthropic's valuation.

If only accredited operators are allowed, which is the only practical way you can enforce this, being one of the existing accredited operators has tremendous value.

American startups fund themselves according to this mechanism: get a shitton of money so you can capture an entire market with unsustainably low prices, then jack up the prices once you have eliminated all competition, thus making back the loss and then some. Uber is the textbook example of this.

The valuation of these companies is, like the value of all stock, an estimation of future returns. And therefore anything that convinces investors that Anthropic will be in a position to jack up the prices at a later date (or at the very least to continue to exist in a competitive market so they can be in a position to do so later) is good for its valuation.

The value of a company is not the quality of its product. It's its ability to make profits.

I still don't think this works out. The total addressable market for AI services in a regulated innovation-skeptical licensing regime is (to within an order of magnetude) $500 billion. The total addressable market in a world where Anthropic is free to pursue unregulated recursive self-improvement is ~all economic value in the universe.

If people thought this way, nobody would ever engage in anti-competitive practices because they reduce growth in the long run.

However, people are humans, with time preference and risk aversion. In the long run we're all dead.

Startups are all about converting the-Mystery-Box into $500B-right-now as soon as is not too soon. Which is why the job of a startup CEO is not to make a good product, but to convince investors that the company is so investable that VCs, himself, and his employees (in that order), can dump all their stocks on retail traders at as high a premium as possible.

No company worth infinity dollars would ever IPO, because a company worth infinity dollars doesn't need funding. Risk adjusted, Anthropic's potentially infinite valuation is actually a very finite number, whose best estimation is what's implied by their last round of funding. And that can go up or down based on many factors, including the regulatory landscape and their relative position therein.

What I claim is that they're entrenched enough at this time (but not necessarily in the future) that this is a good time for them to freeze the market and cash out. Fearmongering serves a dual purpose: it makes the product seem more impressive, and could result in an oligopoly, both of which are good for valuation. Which begs the question: if they really thought they were about to make God in a box, why would they sell it to anyone?

Which begs the question: if they really thought they were about to make God in a box, why would they sell it to anyone?

Probably worth the possibility that they aren't going to sell you God in a box, they are going to get you to pay for them building God in a box.

Or at least that's what the villain would be doing if real life were a movie.

(In real life, it's probably worth asking why they have spun up thousands of instances of AGI to solve Navier-Stokes but apparently zero to solve their cash-flow or public perception problems. Surely arbitrarily large instances of AGI could make datacenter construction popular!)

How did this work out for the nuclear industry in the US? This is absurd, the federal government lately is not in a "give a couple operators a monopoly" state, it's in a "destroy any industry that makes the voters angry" state. The idea that this is a positive EV play is plainly just cynicism for the sake of cynicism. These are not companies whose major competitive advantage is app install base, they're companies whose major competitive advantage is billions of dollars in R&D in an industry where catchup mechanics are brutal with overseas competitors that are not at all subject to preferable regulations. There's just no actual way this works out to being a good EV play over pushing the frontier and hoping for a moonshot. Even if all the major players in all the major labs weren't on record for over a decade with these exact same concerns well before it was clear anyone was going to make any money in this industry this would be a completely retarded plan. "establish a regulatory regime where every company that could have plausible fronted the capital to compete has already done so" is not anything like a real cartel. What are they, afraid of netflix's entry into the sphere?

What are billions of dollars of R&D worth if they give you no moat?

I think you are wrong, that models are commodities, and that market penetration is all that matters. Along with the hardware.

And if the catchup is certain, and you were unfortunate enough to be the first mover and first spender, then you can either take your chances trying to get a lot more money to try again, or you get the government to ban the people that undercut your margins. Preferably both.

"establish a regulatory regime where every company that could have plausible fronted the capital to compete has already done so" is not anything like a real cartel

It'll maybe be enough to save Mistral.

More comments

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats