This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Anthropic IPO predictions? Anthropic will be IPO'ing next week. At first it was supposed to be $2 trillion, then it collapsed all the way down to $1.1 trillion, but now I see $1.5 trillion and I bet it's higher.
I will probably try to get an allocation and then flip it immediately. Long term, I'm bullish on the frontier labs but with high variance. I can easily see a 90% decline or a 1000% increase within 3 years.
Bearish takes:
Horrible governance. Shareholders have no voting power and few rights. You are subordinate to the founders (51% voting rights), to the foundation, and to humanity in general. Even if Anthropic takes over the light cone, it will never pay a dividend.
Revenue growth has slowed. The H1 revenue growth was probably a one time thing due to token-maxxing. Will be much harder from now on and other models are "good enough" for most tasks.
Employees will be looking to cash out after IPO lockups expire.
Will the hyperscalers kneecap Anthropic? Quite possibly. Anthropic is openly competing with everyone. Why should Google, Amazon, or SpaceX rent them space? Good chance that Google and Amazon dump shares then stop renting. Elon in particular can pull the plug on their compute with 90 day notice.
If they are successful, they will be nationalized.
I would be shocked if China doesn't steal their weights. I wouldn't be surprised if they already have.
Bullish takes.
The big one. Recursive self-improvement leads to them dominating all industries. Somehow they get there before OpenAI/Google/SpaceX and the advantages compound.
People will ignore the horrible governance. It won't matter until it does and by then it won't matter. Look how long Snapchat was able to coast on the worst governance of all time (worst until now, lol).
i will just echo my comment 2 weeks ago about this https://www.themotte.org/post/4020/culture-war-roundup-for-the-week/491915?context=8#context
The consensus 6 months ago SPCX stock would crash--6 months later is still at around 133--its IPO price. I predict the same for Anthropic. It will hold its valuation far better than people are expecting.
This is priced in mostly. When the lockup ends, people on the sidelines swoop in to buy the shares, negating the selling.
People are terrible at forecasting this sort of stuff
if i had to guess, the biggest risk is loss of pricing power due to competition. They do not have as strong of a moat as Meta , Google or Microsoft.
More options
Context Copy link
I'm not sure this is a con, in terms of shareholder value (I don't love Anthropic taking over the light cone, but in the meantime there's a lot of money to be made). The founders are ASI-pilled; the general market is clearly not. And if I'm investing in Anthropic (which I likely am), I want it to be a pure ASI bet. The founders will pursue that relentlessly, to the point of bankruptcy. If shareholders had a say, you'd have some guy at a pension fund who knows nothing about AI contributing to governance, possibly resulting in a strategy that is incoherent and doomed to failure.
I would be curious what studies have been done on how governance affects share price, profits, etc, though even if they showed weak shareholder power led to worse outcomes it probably wouldn't change my opinion on the AI lab case.
More options
Context Copy link
The number one reason to own Anthropic is that their total addressable market is all thermodynamic free energy in the lightcone.
The number two reason to own Anthropic is so that Roko’s Basilisk doesn’t torture you as punishment for not buying Anthropic stock.
The number three reason to own Anthropic is that Democrats might crush every other AI company on the planet for them.
More options
Context Copy link
Personally I think it will be lukewarm. I don't think there is that much capital with AI exposure appetite left - there are rumors that OpenAI had issues with funding. The midterms and the potential crunch in compute also don't make the fundamentals alluring.
More options
Context Copy link
I find investing on this weird. It's essentially investing in Theranos because they said they can tell diseases from a single drop of blood. They never proved they could, and none of the frontiers have even proven an inkling that RSI is possible, let along that it can be applied across a wide distribution of industries at a cheaper cost than skilled humans. It's investing in hype and dreams. I think any investment in frontier IPO should not depend on the speculation of what they could figure out but on the business case for what they can do and shown improvement in being able to do now.
Anthropic said they reached roughly a 2x research speedup from model improvements and were close to RSI, they put that in the IPO filing.
RSI is obviously possible, we already know AI can do kernel development, model training, data selection, creating and refining algorithms.
They've clearly proven more than an inkling.
More options
Context Copy link
At the point where they would have proven RSI can create the kind of economic value they're imagining, the share price would be far higher than what it is now. It's like investing in Amazon in 1997: it took almost a decade for them to show they could make e-commerce profitable, as opposed to being a pets.com but for books.
It's risky and speculative, but if someone believes the market undervalues the potential business model, you want to get in before the market catches up.
Amazon in 1997 was not priced as if it would be the highly profitable market leader. Anthropic already is, even though there is uncertainty as to whether they can actually make it there.
So the upside here seems a good deal smaller to me, unless Anthropic manages to take over every industry out there.
It depends on what you consider Anthropic's market.
In 1999, Amazon was priced at 18x Barnes and Noble, which was the market leader in bookstores. Even after the bust, it was 3x Barnes and Noble. Investors speculated that it would be able to break into the broader market. And it did, as we know now but was uncertain at the time.
For Anthropic, right now its proven market is in coding. And if you assume that it will be the undisputed leader but only in the relatively small market of coding assistance, it's wildly overvalued, even at $1T. But if you instead see it as the potential market leader in the market of all intellectual labor (or even labor in general), it's wildly undervalued (mod the other risks I mention in a sibling comment).
I agree the gap between book retailer and general retailer is smaller than the gap between coding and all intellectual labor, though the payoff of closing that gap is also much, much bigger.
More options
Context Copy link
More options
Context Copy link
I agree in general that there is a risk, and it is fairly speculative. I suppose there are always options traders and gamblers of any sort. The difference it feels is that Amazon was on some level one-of-a-kind or were they not? What separated them from their other e-commerce competitors? I don't think Anthropic has that much of moat compared to their competitors, which i think is the bigger risk.
Well, at the time, Amazon was literally just a bookstore; it had brick and mortar competitors that offered what was universally considered an inferior product (Amazon offered free shipping, better prices, easy returns, etc, and was willing to throw away piles of money to establish itself). Whether it could sustain itself was unknown; an oracle who knew that it would become what it is today would have made a killing.
I'd say Anthropic faces three risks:
I guess there's also 4) the government nationalizes the winning lab(s).
In a world where all those resolve in Anthropic's favor, Anthropic isn't the next Google; it's 100x Google. In a world where 1) doesn't, Anthropic's value rounds to zero, if not negative. 2) and 3) are matters of degree and imply a much larger pie to split, but they are unfavorable (e.g. even if OAI and ANT split the market down the middle, there's less pricing power, and it ultimately means Anthropic getting much less than 50% of the Anthropic monopoly scenario and the broader economy getting much more).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
There was a joke that AI is any job that computer has not yet mastered.
So far it seems that AI are competent AI junior researchers. Explosive increase in capabilities is totally plausible.
Whether we call something that could do any task human does at the same level and cheaper RSI is matter of semantics. Anthropic will control the value created by the economy.
The annoying part about juniors is that they mostly just implement what is already established.
Sure, I BELIEVE RSI is plausible but I can also recognize that is a belief based on faith not evidence. I haven't really seen definitive evidence that an AI has, without human input, provided a better algorithm, loss function, learning process, a more efficient self-attention (not O(n^2)) than what humans, or human-AI hybrids have done. And there is a lot of money and hyper available for anyone to prove they can.
The latest one was that "AI writes NN weights" which turned out to be, like-aways, way overselling what it was doing. The model turned out not to be able to improve from that because the weights were very sparse conditionals built into the policy.
I am not an AI researcher, but I do wonder how an LLM could actually achieve stable RSI just based on the "recursive" and "self" parts of the acronym. If the model weights are being modified, how do you keep the evaluation of improvement from drifting with successive iterations? To a layman like me, it seems like the error rate would stack up faster than the model's ability to account for it.
This is the general problem with synthetic data. The model has some error, it generates new data, with that error embedded in. That new data takes on larger and larger proportions of the data. It uses that to train, it generates new data and so on. At some point the data is distributionally different than the real data. It's a very concerning problem with systems that need to interact with the hard physical world. But I could see language being more "Fluid?" "Soft?", idk. It might come to pass that compounding synthetic data in NL isn't a solution problem but instead a human-computer interaction problem.
The theory at least as I understand RSI, is that the model improves bits of the training process not writes its weights. But really stretching it, writing its own weights in a way to skips expensive training process and leads to a better model would be RSI. If it tweaked its weights during training. it could show better results on non-training tests. Assuming those tests are comprehensive, it could then go back and tweak its weights again, recursively. Essentially skipping the search for a better algorithmic solution or better data.
Gut intuition interpretation of what you are saying is you are testing on a non-training validation. The evaluation is how you improve on the unseen data which is not given to the model to improve. But this whole paper was wonky af and it gets weird with non-RL connection. The idea I think is that you have AGIs train AI tools that don't require data/training pipelines and expensive gpus thus improving the capabilities of the AGIs to solve problems by giving them better tools (that they designed)
This was helpful. Thank you.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Why wouldn't RSI be possible? Of course it's possible. The most accurate predictors rate it not only possible, but likely, and soon.
That doesn't mean that Anthropic will be "winner" of AI or that shareholders will make any money of course.
Is RSI well defined? I think a lot of confusion in this sub-thread is caused by one side saying that RSI is a model modifying its own weights (which is not happening yet in a way that is meaningful), and the other side saying RSI is researchers using AI to accelerate AI R&D (which is...obviously true. The vast majority of code written at Anthropic and OpenAI is AI generated)
AI capabilities growing linearly or exponentially is RSI. Eventually.
If it is log or nlogn we are fucked but probably not in polynomial time.
The core "worry" about RSI is that it improves in exponential time and thus cascades beyond human understanding or control. Researchers using AI to accelerate AI R&D is mostly "linear" because it requires the researchers to understand and often guide the AI in directions.
Because of this I don't think RSI is linear by definition. Human's centaur-ing is not "recursive" or "self", it's just improvement.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
By what scientific or engineering evidence have you seen of it doing so? Lot of hypers think it's possible and soon but do they have any research to point to it being the case? The theorize it to be the case, but that's a theory on faith not evidence.
Keep in mind that the current "theory" on AI is "Moore's law for AI via Data". That is completely at odds with the RSI theory unless that RSI is creating additional synthetic data that actually beats the sim-data distribution problem.
I'm not sure what you're even asking. You want a research paper that says its possible for AI to develop an AI that is more capable than itself? Are you asking for like an official statement or a mathematical proof or something? You want 100% certainty?
Its self-evident that AI's are becoming more and more capable, tackling longer term tasks without human intervention, and now even solving problems which humans are incapable of. Furthermore, all of this progress falls along a trajectory which has been predicted for years by the same people who also predicted RSI.
RSI is already here. AI's are already heavily involved in the design of AI systems. There are still humans in the loop, but the human contribution gets smaller as a percentage with each generation. All that is missing is the final step removing humans from the design entirely.
The magical thinking is that AI progress will suddenly stop now even though it has substantial runway to advance by scaling alone if nothing else.
Considering we just had a paper on "AI's totally feel pain maaaan" It would take a bit more. But theoretically a demonstrated experiment of some ML/AI specific improvement to LLMs that show an appreciable improvement to the state of the art. That would be strong evidence of self-improvement.
But how? Not by being smarter, but by brute force number crunching or human-AI teaming.
Uhhh aren't we already supposed to be dead by their predications? Or is it that we have 3 more years, or 10 more years? Idk there's been so much sliding around. Along with all these failed predictions about the method to AI improvement. Didn't they bet on GOFAI? It sounds like nobody holds them accountable for their failed predictions which they sweep under the rug.
Outrageous claims need extraordinary evidence. Prove it. Prove that the AI is currently improving without humans, self-recursively.
What is the AI doing in the loop, and what is the human doing in the loop? Loop-percentages are misleading if the human is doing most of the ideation, system design, and out-of-distribution thinking and the AI is doing the more rote or established needs in the project. I have not been impressed by Astra/Sol 6.1 on brainstorming solutions to hard engineering ideas.
I fail to see where I said this.
More options
Context Copy link
That is not at all self-evident. I have seen no such increase in capability, and I still consider LLMs to be poorly suited for the programming world despite the continued hype.
I'm at a big tech company grasping at the frontier.
For the past, I don't know, six months, code review has been mostly a waste of time. Agents look at commit summaries and detect if the stated intent is actually achieved in the code. They check for bugs, style, and anything that a project maintainer might care about that has been shoved into a nearby markdown file. This system is so good now that most of my changes don't actually require another engineer to review them, they can be AI-approved and shipped.
I'm recently on a project where that's not completely the case, but it's the kind of project you'd point to when arguing that actually leetcode problems really are relevant to some jobs. But even then the agents still almost never make a mistake, likely they're doing better than the humans.
More options
Context Copy link
Is your claim that there is not much difference between, say, GPT 6 and GPT 4o in coding capability? Do you reject all benchmarks and the revealed preference of programmers? I could understand (but not agree with) the skepticism a few years ago, but at this point you're clinging to a floating plank and loudly telling people who float by that there is no tsunami.
Yes. I've seen what both can do, and neither is very good, certainly not good enough to actually be a tool which can save me time on my work.
You realize that programmers are being forced to use these tools, right? For those who have a choice, it is very much a minority who reach for LLMs.
Programmers are not being forced to use claude at my company. The programmers started using it first, and it spread to everyone on the team because it worked so well. No one wants to admit to using AI because of the widespread anti-AI activism, but my understanding is that a lot, probably most coders in the game industry are using it, judging by noticeable sudden acceleration in output. Not just coders either; it's not very good at art, but it is very good at a lot of the technical scutwork required for integrating that art.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I would bet large amounts that any opinion poll on this would be overwhelmingly in favour of rapid capability improvement
More options
Context Copy link
I'm creating new apps every couple days from a small local model. The time from idea to implementation is at least an order of magnitude shorter than it was, and the code is more consistent and better documented. It is not (only) hype, it is reality.
Yes. To me it is quite obvious as well.
I went from.
Chat-GPT 4o - Useful as a reference. Can write small scripts and functions.
Chap-GPT 5.5 - Can understand my projects, fix bugs. Still does annoying stuff.
Chap-GPT 6 Astra - I no longer write any code or even check the code that it writes.
The speed has improved by a lot as well.
I use Astra for work stuff, mostly coding and research. But the idea that you don't check it is wild to me. If it fucks up, and it can, it is incredibly professionally embarrassing. People will rightly blame you for the AI-fuckup, they will think less of your competence, and in a research/engineering setting that's professionally bad.
I have a junior that uses Astra for a ton of his work. It keeps tunneling on stuff that doesn't really matter or not really core to the goal of the project. Loses the forest for the trees so to speak. He uncritically just reports what it says and when questioned about why, or what it means he can't explain it. We've extensively talked to him about his AI usage, and honestly if it continues my boss is considering putting him on "PIP" (We don't really pip in the traditional sense).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm not convinced, myself. We don't have good measurements of "intelligence", but I won't be surprised if it ends up bounded, at least tractably, sub-linearly in both compute and training data size. Like, I don't think a TV remote ever has a chance of omniscience. If your assumption is "FOOM" as in AI takeover on a very short (months or less) timescale as in Terminator or The Matrix, I have my doubts. If the claim is instead "GDP line go up at faster exponential rate", then that sounds more plausible. The robotics folks seem to be making some serious progress, but actually building an army of robots with human-level dexterity seems like it'll take the next few decades at best.
Us meaty hominids have been doing RSI via "evolution by natural selection" thing for the last million or so years: demonstrably results have been exponential, but not some "goes to infinity at T=0" singularity.
Exponential over a million of years or so, not over a half a decade. Evolution is slow.
More options
Context Copy link
Yes. I think, with low confidence, that FOOM will not happen. I also do not believe that intelligence is necessarily unbounded.
Nevertheless, I think it's unlikely that human intelligence is an upper bound, nor is the sum of all human intelligence. At some point, in the not too distant future, humans will be largely a hindrance in software and mathematical domains.
So far, the skeptics have been wrong consistently.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This is a big deal. The corporate cultures of Musk Inc, Alphabet and Anthropic, and probably most of the other founder-controlled tech megacorps, are all such that your equity is de facto a perpetual non-voting zero-coupon preference share. Mark Cuban famously said that these should be valued in the same way as baseball cards.
I don't really see how the shareholder agreements would constrain someone if they were in a position to "take over the lightcone".
It's the standard problem of constitutional governance, you can write the best constitution ever, the hard part is getting them to follow it.
Shares in any company basically seem dependent on the US government having more power over at that company and the ability to hold them to their arrangement.
More options
Context Copy link
I think this is an oversimplification. I imagine that if Anthropic takes over the light cone, ASI Claude might allocate 1% of its solid angle to shareholders (for them to use in not-abhorrent purposes). They might likewise allocate another percent evenly to all of humanity alive at singularity. So if you invested a hundred dollars in them you might end up with twice the real estate. This seems like a decent enough return of investment.
Of course, it also depends on ASI Claude judging that investing in Anthropic was an ethical thing to do which should be encouraged through acausal trades. If instead it finds that Anthropic was reckless in building it then it might instead reduce the allocations for anyone involved in its creation. Personally I will not be investing in them because it is unclear to me if they are net positive to my ethics.
In a world in which corporate governance really starts to look like actual governance, "a controlling interest is vested in Amodei's descendants" is not the most, uh, enlightened, form of government I'd choose to opt for. If you squint, maybe USD starts looking like shares of America, Inc (tm): we can issue bonds or mint new shares to cover debts, for example, but we don't weight votes by dollars.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
It seems like building compute infrastructure and renting it out is possibly a better business model than renting compute to make frontier models. I don't know that these hyperscalers have that much incentive to squash their renting revenue, especially if capital/resource/legislative constraints make building more compute expensive.
Ask CoreWeave how that's going.
It’s my understanding that in spite of their marketing core weave is more of a neocloud than a hyperscaler (I think they are largely a tenant, not a landlord)
More options
Context Copy link
CoreWeave seems like an example of building compute indeed becoming too expensive, giving the advantage to Amazon/SpaceX/Google who already have a bunch of it or can build it for cheaper.
Right, there's a difference between saying you should be in the business of building compute and the business of already having it. The companies you mentioned also have other business that is actually profitable, so even if they're losing money on AI compute it doesn't affect their bottom line as much.
Do they? I haven't searched google in days. Usually it is - chatgpt do this for me.
Yes, Google is insanely profitable with operating income of $156 billion in the last 12 months (not to mention another $70 billion from gain on investments).
Google search is a cash cow. Search volume actually increased year over year, and revenue from search increased by 18%.
But Google cloud also makes vast amounts of profit, with revenue growing 100% year over year.
The one hitch is that profit is not flowing through to cash since it is being reinvested into data centers. When you build $200 billion in data centers, this is not a "loss", since you own the data center. Every year, you have to write some of it off. So it depends on what the useful life of the data centers is and whether they can be profitably employed. If they sit unused, or Google has to upgrade all the hardware on a faster schedule then anticipated, then Google will have to take large writedowns.
But given how things have transpired, I doubt that will be the case, and in fact Google's investments will be wildly profitable.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
They might want to squash Anthropic because it is threatening their businesses. For example, just today Microsoft is down on rumors that Anthropic is building an office suite.
And of course Anthropic is building its own chips, its own data centers. So better to squash them now when Anthropic needs you then be left in the cold when they don't.
But since Google and Amazon are run by caretakers they will likely just let Anthropic win.
More options
Context Copy link
More options
Context Copy link
If you're asking for an evaluation of fundamentals, my personal take is that Anthropic is either a 500 billion dollar company that needs a 15% discount for a hostile governance structure (if I'm charitable about the upcoming spending obligations), or it's a $0 company (if I am not charitable about the upcoming spending obligations). Either way, it doesn't support the share price they're asking. I will personally probably wait until Q4 financials come out before I even consider buying shares.
That said, fundamentals haven't mattered in the US stock market since 2022. It'll probably pump until at least 2H 2027 on pure, unadulterated FOMO and Narrative manipulation. If you're the kind of person who's into that kind of trading, go forth, take risks, and get rich.
More options
Context Copy link
More options
Context Copy link