This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
AI X-Risk Goes Mainstream?
Jacob Coxon:
116,400,000 views, 634,000 likes.
Evan Hubinger (head of alignment at Anthropic):
34,000,000 views, 51,000 likes.
None of this is substantially different from what certain AI industry insiders have been saying for 5-ish years now, but it's clear from the replies, quote-tweets, and engagement statistics that most people are hearing about this for the first time. Yes, there was clearly some coordination with friendly media prior to the announcement given the almost simultaneous WSJ article (whereas news outlets were initially reluctant to publish anything on the unexpected news of OpenAI solving a Millennium Prize Problem). One can take the maximally conspiratorial position that this is a 4D bankshot by Democrats to end tech innovation forever, but it was going to break through eventually.
In other words, Coffeezilla is now on the case.
Greater than 10% of apocalypse over the next decade? That is such spectacular risk tolerance that I have a hard time believing this 10% number is genuinely held, which could just be that the people who believe in the "10%" chance do not really know what it means to take a 10% risk.
It's a bullshit number, but what it really means is "I take this as seriously if not more than nuclear Armageddon" which, as we know around these parts, is an opinion popular in Silicon Valley and even more so at Anthropic.
And not to sound like a broken record, but what is really really means is "please please regulate us so we can get an oligopoly and cash out our insane valuations before China eats our lunch"
You know, this kind of makes sense for when Anthropic lobbies against open weights, or against Chinese model use, or in favor of chip export restrictions, but I don't really see how regulating frontier capabilities research helps Anthropic's valuation.
If only accredited operators are allowed, which is the only practical way you can enforce this, being one of the existing accredited operators has tremendous value.
American startups fund themselves according to this mechanism: get a shitton of money so you can capture an entire market with unsustainably low prices, then jack up the prices once you have eliminated all competition, thus making back the loss and then some. Uber is the textbook example of this.
The valuation of these companies is, like the value of all stock, an estimation of future returns. And therefore anything that convinces investors that Anthropic will be in a position to jack up the prices at a later date (or at the very least to continue to exist in a competitive market so they can be in a position to do so later) is good for its valuation.
The value of a company is not the quality of its product. It's its ability to make profits.
I still don't think this works out. The total addressable market for AI services in a regulated innovation-skeptical licensing regime is (to within an order of magnetude) $500 billion. The total addressable market in a world where Anthropic is free to pursue unregulated recursive self-improvement is ~all economic value in the universe.
If people thought this way, nobody would ever engage in anti-competitive practices because they reduce growth in the long run.
However, people are humans, with time preference and risk aversion. In the long run we're all dead.
Startups are all about converting the-Mystery-Box into $500B-right-now as soon as is not too soon. Which is why the job of a startup CEO is not to make a good product, but to convince investors that the company is so investable that VCs, himself, and his employees (in that order), can dump all their stocks on retail traders at as high a premium as possible.
No company worth infinity dollars would ever IPO, because a company worth infinity dollars doesn't need funding. Risk adjusted, Anthropic's potentially infinite valuation is actually a very finite number, whose best estimation is what's implied by their last round of funding. And that can go up or down based on many factors, including the regulatory landscape and their relative position therein.
What I claim is that they're entrenched enough at this time (but not necessarily in the future) that this is a good time for them to freeze the market and cash out. Fearmongering serves a dual purpose: it makes the product seem more impressive, and could result in an oligopoly, both of which are good for valuation. Which begs the question: if they really thought they were about to make God in a box, why would they sell it to anyone?
Probably worth the possibility that they aren't going to sell you God in a box, they are going to get you to pay for them building God in a box.
Or at least that's what the villain would be doing if real life were a movie.
(In real life, it's probably worth asking why they have spun up thousands of instances of AGI to solve Navier-Stokes but apparently zero to solve their cash-flow or public perception problems. Surely arbitrarily large instances of AGI could make datacenter construction popular!)
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link