This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
AI X-Risk Goes Mainstream?
Jacob Coxon:
116,400,000 views, 634,000 likes.
Evan Hubinger (head of alignment at Anthropic):
34,000,000 views, 51,000 likes.
None of this is substantially different from what certain AI industry insiders have been saying for 5-ish years now, but it's clear from the replies, quote-tweets, and engagement statistics that most people are hearing about this for the first time. Yes, there was clearly some coordination with friendly media prior to the announcement given the almost simultaneous WSJ article (whereas news outlets were initially reluctant to publish anything on the unexpected news of OpenAI solving a Millennium Prize Problem). One can take the maximally conspiratorial position that this is a 4D bankshot by Democrats to end tech innovation forever, but it was going to break through eventually.
In other words, Coffeezilla is now on the case.
Greater than 10% of apocalypse over the next decade? That is such spectacular risk tolerance that I have a hard time believing this 10% number is genuinely held, which could just be that the people who believe in the "10%" chance do not really know what it means to take a 10% risk.
It's a bullshit number, but what it really means is "I take this as seriously if not more than nuclear Armageddon" which, as we know around these parts, is an opinion popular in Silicon Valley and even more so at Anthropic.
And not to sound like a broken record, but what is really really means is "please please regulate us so we can get an oligopoly and cash out our insane valuations before China eats our lunch"
You know, this kind of makes sense for when Anthropic lobbies against open weights, or against Chinese model use, or in favor of chip export restrictions, but I don't really see how regulating frontier capabilities research helps Anthropic's valuation.
If only accredited operators are allowed, which is the only practical way you can enforce this, being one of the existing accredited operators has tremendous value.
American startups fund themselves according to this mechanism: get a shitton of money so you can capture an entire market with unsustainably low prices, then jack up the prices once you have eliminated all competition, thus making back the loss and then some. Uber is the textbook example of this.
The valuation of these companies is, like the value of all stock, an estimation of future returns. And therefore anything that convinces investors that Anthropic will be in a position to jack up the prices at a later date (or at the very least to continue to exist in a competitive market so they can be in a position to do so later) is good for its valuation.
The value of a company is not the quality of its product. It's its ability to make profits.
I still don't think this works out. The total addressable market for AI services in a regulated innovation-skeptical licensing regime is (to within an order of magnetude) $500 billion. The total addressable market in a world where Anthropic is free to pursue unregulated recursive self-improvement is ~all economic value in the universe.
If people thought this way, nobody would ever engage in anti-competitive practices because they reduce growth in the long run.
However, people are humans, with time preference and risk aversion. In the long run we're all dead.
Startups are all about converting the-Mystery-Box into $500B-right-now as soon as is not too soon. Which is why the job of a startup CEO is not to make a good product, but to convince investors that the company is so investable that VCs, himself, and his employees (in that order), can dump all their stocks on retail traders at as high a premium as possible.
No company worth infinity dollars would ever IPO, because a company worth infinity dollars doesn't need funding. Risk adjusted, Anthropic's potentially infinite valuation is actually a very finite number, whose best estimation is what's implied by their last round of funding. And that can go up or down based on many factors, including the regulatory landscape and their relative position therein.
What I claim is that they're entrenched enough at this time (but not necessarily in the future) that this is a good time for them to freeze the market and cash out. Fearmongering serves a dual purpose: it makes the product seem more impressive, and could result in an oligopoly, both of which are good for valuation. Which begs the question: if they really thought they were about to make God in a box, why would they sell it to anyone?
How did this work out for the nuclear industry in the US? This is absurd, the federal government lately is not in a "give a couple operators a monopoly" state, it's in a "destroy any industry that makes the voters angry" state. The idea that this is a positive EV play is plainly just cynicism for the sake of cynicism. These are not companies whose major competitive advantage is app install base, they're companies whose major competitive advantage is billions of dollars in R&D in an industry where catchup mechanics are brutal with overseas competitors that are not at all subject to preferable regulations. There's just no actual way this works out to being a good EV play over pushing the frontier and hoping for a moonshot. Even if all the major players in all the major labs weren't on record for over a decade with these exact same concerns well before it was clear anyone was going to make any money in this industry this would be a completely retarded plan. "establish a regulatory regime where every company that could have plausible fronted the capital to compete has already done so" is not anything like a real cartel. What are they, afraid of netflix's entry into the sphere?
What are billions of dollars of R&D worth if they give you no moat?
I think you are wrong, that models are commodities, and that market penetration is all that matters. Along with the hardware.
And if the catchup is certain, and you were unfortunate enough to be the first mover and first spender, then you can either take your chances trying to get a lot more money to try again, or you get the government to ban the people that undercut your margins. Preferably both.
It'll maybe be enough to save Mistral.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link