This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Anthropic IPO predictions? Anthropic will be IPO'ing next week. At first it was supposed to be $2 trillion, then it collapsed all the way down to $1.1 trillion, but now I see $1.5 trillion and I bet it's higher.
I will probably try to get an allocation and then flip it immediately. Long term, I'm bullish on the frontier labs but with high variance. I can easily see a 90% decline or a 1000% increase within 3 years.
Bearish takes:
Horrible governance. Shareholders have no voting power and few rights. You are subordinate to the founders (51% voting rights), to the foundation, and to humanity in general. Even if Anthropic takes over the light cone, it will never pay a dividend.
Revenue growth has slowed. The H1 revenue growth was probably a one time thing due to token-maxxing. Will be much harder from now on and other models are "good enough" for most tasks.
Employees will be looking to cash out after IPO lockups expire.
Will the hyperscalers kneecap Anthropic? Quite possibly. Anthropic is openly competing with everyone. Why should Google, Amazon, or SpaceX rent them space? Good chance that Google and Amazon dump shares then stop renting. Elon in particular can pull the plug on their compute with 90 day notice.
If they are successful, they will be nationalized.
I would be shocked if China doesn't steal their weights. I wouldn't be surprised if they already have.
Bullish takes.
The big one. Recursive self-improvement leads to them dominating all industries. Somehow they get there before OpenAI/Google/SpaceX and the advantages compound.
People will ignore the horrible governance. It won't matter until it does and by then it won't matter. Look how long Snapchat was able to coast on the worst governance of all time (worst until now, lol).
I find investing on this weird. It's essentially investing in Theranos because they said they can tell diseases from a single drop of blood. They never proved they could, and none of the frontiers have even proven an inkling that RSI is possible, let along that it can be applied across a wide distribution of industries at a cheaper cost than skilled humans. It's investing in hype and dreams. I think any investment in frontier IPO should not depend on the speculation of what they could figure out but on the business case for what they can do and shown improvement in being able to do now.
There was a joke that AI is any job that computer has not yet mastered.
So far it seems that AI are competent AI junior researchers. Explosive increase in capabilities is totally plausible.
Whether we call something that could do any task human does at the same level and cheaper RSI is matter of semantics. Anthropic will control the value created by the economy.
The annoying part about juniors is that they mostly just implement what is already established.
Sure, I BELIEVE RSI is plausible but I can also recognize that is a belief based on faith not evidence. I haven't really seen definitive evidence that an AI has, without human input, provided a better algorithm, loss function, learning process, a more efficient self-attention (not O(n^2)) than what humans, or human-AI hybrids have done. And there is a lot of money and hyper available for anyone to prove they can.
The latest one was that "AI writes NN weights" which turned out to be, like-aways, way overselling what it was doing. The model turned out not to be able to improve from that because the weights were very sparse conditionals built into the policy.
I am not an AI researcher, but I do wonder how an LLM could actually achieve stable RSI just based on the "recursive" and "self" parts of the acronym. If the model weights are being modified, how do you keep the evaluation of improvement from drifting with successive iterations? To a layman like me, it seems like the error rate would stack up faster than the model's ability to account for it.
This is the general problem with synthetic data. The model has some error, it generates new data, with that error embedded in. That new data takes on larger and larger proportions of the data. It uses that to train, it generates new data and so on. At some point the data is distributionally different than the real data. It's a very concerning problem with systems that need to interact with the hard physical world. But I could see language being more "Fluid?" "Soft?", idk. It might come to pass that compounding synthetic data in NL isn't a solution problem but instead a human-computer interaction problem.
The theory at least as I understand RSI, is that the model improves bits of the training process not writes its weights. But really stretching it, writing its own weights in a way to skips expensive training process and leads to a better model would be RSI. If it tweaked its weights during training. it could show better results on non-training tests. Assuming those tests are comprehensive, it could then go back and tweak its weights again, recursively. Essentially skipping the search for a better algorithmic solution or better data.
Gut intuition interpretation of what you are saying is you are testing on a non-training validation. The evaluation is how you improve on the unseen data which is not given to the model to improve. But this whole paper was wonky af and it gets weird with non-RL connection. The idea I think is that you have AGIs train AI tools that don't require data/training pipelines and expensive gpus thus improving the capabilities of the AGIs to solve problems by giving them better tools (that they designed)
This was helpful. Thank you.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link