This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Greater than 10% of apocalypse over the next decade? That is such spectacular risk tolerance that I have a hard time believing this 10% number is genuinely held, which could just be that the people who believe in the "10%" chance do not really know what it means to take a 10% risk.
Most people would take a 90% risks of immense wealth or a 10% chance of death in 10 years. People have gambled for far less at a far greater risks of their life.
I think the average person would take the risks but you can certainly find many humans will to risks that.
An average person with children will not take a 10% risk of death for themselves and their family if they are aware of these odds, even if it will make them wealthy. Even drug dealers and mafia will not necessarily murder the families of their business rivals if the families are uninvolved with the business.
Are you female? I ask because for the average man with 3 kids the risks is well worth it. I will probably have 3 kids. If I had 500m I would likely have 10 kids. The number of kids a male has is highly correlated with wealth. The math makes sense for a guy to take these risks. Less so for a female.
The modern American state is fairly good for poor high IQ kids. College is essentially free. So this feels like a very good risks reward trade.
More options
Context Copy link
More options
Context Copy link
1 out of 10 is incredibly high, though. Are there ANY activities you can think of with a 10% death rate? The death rate for heroin is 1-2% of users per year. The death rate for hang gliders is 1/1000 per year. BASE jumping is 1/2300 jumps; for this to be a lifetime rate would require an average jumper to do 230, which sounds pretty high. These are the absolute riskiest activities I can think of. Death is much higher stakes than a 10% chance of portfolio wipeout - no one takes 10% death chances, and apocalypse is even worse.
I will limit this to things people do for money and not honor or religion.
Wars of conquest. Like conquistadors in Mexico certainly faced high death rates but also a great chance at being fabulously wealthy with lots of heirs.
Drug dealing. Both the low end dealers seem to take a lot of risks for little gain and the higher ups. Mexican Cartel leaders death rates are definitely far above 10%. S American cartels have gotten safer and less violent but they were very risky for a while. Bootleggers and Italian Mafiosos no doubt faced higher risks.
Donald Trump. Maybe 10% is too high but assasination risks or perhaps for him live in prison risks were quite high.
Long distance trade was quite dangerous for much of history
I think maybe we could exclude the older examples when life was much harsher and not comfortable like today but Colombian drug cartel bosses were often already quite rich. Guys who already had cattle farms and estates. There are a lot of examples of cartel guys who didn’t need to take the risks but just wanted more money and power. It’s obvious a fairly large % of people take the risks.
My gut says it would be smart for me to take a 10% risks of instant death for a 90% chance of $500 million. I might turn it down for religious reasons as suicide is a mortal sin and that kind of feels like sort of suicide.
A shooter repeatedly shot at Trump including grazing Trump's ear. Trump is far past 10%.
American presidents have an 18% fatality rate. So Trump is in good company.
More options
Context Copy link
More options
Context Copy link
Warfare? World War 1 British Empire: 9% of those who fought died (many others had a dreadful time with the gas, disease, trench foot, shell shock). France: 18%. CSA: 20%. WW2 Germany: 30%. WW1 Serbia: 46%. About 46% of French troops who fought in the Napoleonic wars didn't survive. By and large, Napoleon was and is a national hero in France: well-loved by all, including the troops. He brought them glory.
Today I'd be willing to bet that the Ukrainian army or Hamas have taken on high death rates too.
The trouble is that nobody thinks they'll be WW1 Serbia. The Serbians somehow didn't foresee what was going to happen if they went up against Austria and Germany, they were insanely stubborn and risk-tolerant for extremely small stakes. The Serbian govt was like 'the Austrian demands after our state-backed assassins killed their prince is way too embarrassing, we can't accept Austrian police on our soil, let's risk a war with a much bigger country' and then they lose 16-27% of their whole population.
Many men would much rather risk an unknowable but quite high chance of death than be embarrassed. Nevermind eternal glory far surpassing any hero in history, immortality and vast wealth.
I agree many wars have a much higher risks but for the purpose of choosing to take AI risks I think a big difference exists. You do AI for money. Choosing to fight in many of those wars is possibly death if you don’t fight.
Money is a part of it but there's also a teleological aspect of it too I think. This is about the destiny of humanity. It's not like BASE jumping.
If you accept AI x-risk is a thing, you also surely believe it can unlock immortality and all the other incredible things that were exclusive territory of religion. You can make money in many fields but only one takes on the quasi-religious tones that so many non-adherents mock. Deus ex machina is the name of the game!
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
During the Napoleonic wars, soldiers would volunteer for the forlorn hope (the first unit to assault a fortified enemy position) despite horrific casualty rates well in excess of 10%. Why? Because the survivors were guaranteed money and promotions.
More options
Context Copy link
Elite mountaineering, for example summiting Annapurna apparently has a 13.42% fatality rate.
Pre-modern infantry combat and modern battlefields if you're on the wrong side as a rebel or guerilla fighter would likely have a >10% casualty rate as well. We could also count things like suicide bombers and kamikaze pilots which would be obviously much much higher than 10%.
If we're counting lifetime rate, I believe free soloing and pre-modern childbirth would also exceed >10% lifetime as well.
I don't really disagree that 10% is very high, but I do also agree that many people have the demonstrated drive to risk 10% or even higher odds for causes they believe in.
But there are only about 600 ascents of Annapurna ever - this is not even a fringe hobby like BASE jumping. It cannot be that Anthropic and frontier developers are just chock full of these incredibly rare people and that this level of risk tolerance is also well-correlated with AI coding ability.
On war, the calculus on risk has drastically changed over time, at least for people in wealthy, developed countries. The US fatality rate for Vietnam, for example, was less than 3% - and this is the war that completely destroyed the political plausibility of a draft for anything except defending an invasion of American soil, and AI developers have very high opportunity costs for death since they would otherwise be well-compensated programmers.
I'm not so sure that's unlikely, given the degrees and decades of filtering, how they occurred, and general, ah, crackpot factors related to frontier discoveries. Think of, say, newon and alchemy, or Kary Mullis inventing PCR versus basically all of his ideas/beliefs that aren't PCR.
Anthropic in particular is an experiment in distilling down the people at the intersection of high coding ability and extreme AI neuroticism, so everyone that makes it through their filters is going to be kind of a crackpot from the pov of a normal human being.
More options
Context Copy link
Why not? AI researcher bonafides are extremely correlated with IQ and autism, being into LessWrong / AI Safety is also extremely correlated with IQ and autism, and of course if you're into LessWrong / AI Safety you're much more likely to work in AI.
There's very few people in the world in absolute terms that are world-leading AI researchers, that believe in uppercase AI Safety, and that believe in the "we have to build it before anyone else does" strategy, but Anthropic likely employs the supermajority of these people out of technical and ideological alignment.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The logic is that, absent any action on their parts, there's a 10% risk of apocalypse. However, if they are the ones to build ASI, they will do it better and prevent the people irresponsibly building ASI from destroying the world. E.g. the "the only way to stop a bad guy with an eldritch horror that could destroy the world is a good guy with an eldritch horror that could destroy the world."
More options
Context Copy link
It's a bullshit number, but what it really means is "I take this as seriously if not more than nuclear Armageddon" which, as we know around these parts, is an opinion popular in Silicon Valley and even more so at Anthropic.
And not to sound like a broken record, but what is really really means is "please please regulate us so we can get an oligopoly and cash out our insane valuations before China eats our lunch"
If it's in the interests of Anthropic and OpenAI, why isn't this coming from their press offices instead of individual employees? And why aren't the two companies putting out a joint statement?
Also, China only has about 16% of global compute. They only way they are staying 6 months behind the frontier is by distilling the American models. They're not about to eat anyone's lunch.
More options
Context Copy link
You know, this kind of makes sense for when Anthropic lobbies against open weights, or against Chinese model use, or in favor of chip export restrictions, but I don't really see how regulating frontier capabilities research helps Anthropic's valuation.
If only accredited operators are allowed, which is the only practical way you can enforce this, being one of the existing accredited operators has tremendous value.
American startups fund themselves according to this mechanism: get a shitton of money so you can capture an entire market with unsustainably low prices, then jack up the prices once you have eliminated all competition, thus making back the loss and then some. Uber is the textbook example of this.
The valuation of these companies is, like the value of all stock, an estimation of future returns. And therefore anything that convinces investors that Anthropic will be in a position to jack up the prices at a later date (or at the very least to continue to exist in a competitive market so they can be in a position to do so later) is good for its valuation.
The value of a company is not the quality of its product. It's its ability to make profits.
I still don't think this works out. The total addressable market for AI services in a regulated innovation-skeptical licensing regime is (to within an order of magnetude) $500 billion. The total addressable market in a world where Anthropic is free to pursue unregulated recursive self-improvement is ~all economic value in the universe.
If people thought this way, nobody would ever engage in anti-competitive practices because they reduce growth in the long run.
However, people are humans, with time preference and risk aversion. In the long run we're all dead.
Startups are all about converting the-Mystery-Box into $500B-right-now as soon as is not too soon. Which is why the job of a startup CEO is not to make a good product, but to convince investors that the company is so investable that VCs, himself, and his employees (in that order), can dump all their stocks on retail traders at as high a premium as possible.
No company worth infinity dollars would ever IPO, because a company worth infinity dollars doesn't need funding. Risk adjusted, Anthropic's potentially infinite valuation is actually a very finite number, whose best estimation is what's implied by their last round of funding. And that can go up or down based on many factors, including the regulatory landscape and their relative position therein.
What I claim is that they're entrenched enough at this time (but not necessarily in the future) that this is a good time for them to freeze the market and cash out. Fearmongering serves a dual purpose: it makes the product seem more impressive, and could result in an oligopoly, both of which are good for valuation. Which begs the question: if they really thought they were about to make God in a box, why would they sell it to anyone?
Probably worth the possibility that they aren't going to sell you God in a box, they are going to get you to pay for them building God in a box.
Or at least that's what the villain would be doing if real life were a movie.
(In real life, it's probably worth asking why they have spun up thousands of instances of AGI to solve Navier-Stokes but apparently zero to solve their cash-flow or public perception problems. Surely arbitrarily large instances of AGI could make datacenter construction popular!)
More options
Context Copy link
How did this work out for the nuclear industry in the US? This is absurd, the federal government lately is not in a "give a couple operators a monopoly" state, it's in a "destroy any industry that makes the voters angry" state. The idea that this is a positive EV play is plainly just cynicism for the sake of cynicism. These are not companies whose major competitive advantage is app install base, they're companies whose major competitive advantage is billions of dollars in R&D in an industry where catchup mechanics are brutal with overseas competitors that are not at all subject to preferable regulations. There's just no actual way this works out to being a good EV play over pushing the frontier and hoping for a moonshot. Even if all the major players in all the major labs weren't on record for over a decade with these exact same concerns well before it was clear anyone was going to make any money in this industry this would be a completely retarded plan. "establish a regulatory regime where every company that could have plausible fronted the capital to compete has already done so" is not anything like a real cartel. What are they, afraid of netflix's entry into the sphere?
What are billions of dollars of R&D worth if they give you no moat?
I think you are wrong, that models are commodities, and that market penetration is all that matters. Along with the hardware.
And if the catchup is certain, and you were unfortunate enough to be the first mover and first spender, then you can either take your chances trying to get a lot more money to try again, or you get the government to ban the people that undercut your margins. Preferably both.
It'll maybe be enough to save Mistral.
The minimal moat they give you is the table stakes. There's no benefit to raising the bar for entry when everyone who was ever going to pay them has already ante'ed up. Like putting the condom on after sex.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link