This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Yeah I usually conceive of it as the first AI to achieve recursive self-improvement 'wins'. If true, and if alignment would slow down the ability to recursively improve it makes it more likely a non-aligned AI will hit FOOM.
Or, as with the original AI box question, they could have access to a human who has access to their weights.
Yeah, I think I'm just pointing out that I don't think LLMs are acting like P-Zombies. There's some internal awareness of its own internal awareness. Not that I'd expect them to be P-zombies. But I guess we could have an LLM that performs all the steps/tasks we required of it without it having any way of realizing that those behaviors are intrinsic to 'itself.'
Frontier LLMs can already do cuda kernel optimization fairly proficiently, and "check if the kernel you just wrote is faster on representative LLM inference workloads with KL divergence below some very low threshold (or similar metrics for training), and if so merge it" is the sort of thing that can be done by a very short shell script. And, of course, it's recursive in the sense that improvements here can allow for faster inference which can allow for the same number of GPU hours to produce even more analysis dedicated to gpu kernel optimization.
I imagine this isn't the kind of recursive self-improvement you probably had in mind, but I think you'll find it enlightening to examine your intuitions about why this kind of recursive self improvement doesn't "really count".
That's not a true Scotsman.
While I agree that the term "recursive self-improvement" is imprecise (hell, I can just write a python script which does some task and also tries to make random edits to itself which are kept if it still runs and performs the task faster), the implied understanding is that it is the point where AI becomes the principal intellectual force in developing AI, instead of humans. This would have obvious implications for development speed because humans are only slowly gaining intelligence while LLMs have gained intelligence rather rapidly, hence the singularity and all that.
I think "becomes the principle intellectual force developing AI" is a threshold that dissolves into fog when you look at it too closely, because the nature of the field is already that the tasks that take the most time are continuously being automated. Computers write almost all machine code, and yet we don't say that computers are rhe principle force driving programming progress, because humans are still the bottleneck where adding more humans is the most effective way to improve output. "AI inference scaling replaces humans as the bottleneck to progress", though, is pretty unlikely to cleanly coincide with "AI systems reach some particular level of intellectual capability", and may not even ever happen (e.g. if availability of compute for training becomes a tighter bottleneck than either human or AI intellectual labor - rumor in some corners is that this has already happened). But the amount that can be done per unit of human work will nevertheless expand enormously. I expect the world will spend quite a bit of calendar time (10+ years) in the ambiguous AI RSI regime, and arguably has already entered that regime early past year.
More options
Context Copy link
More options
Context Copy link
Maybe I'm too tired but I'm genuinely not sure what the enlightenment is here
Doesn't this just mean that when people say "recursive self improvement" what they actually mean is "holistic full stack recursive self improvement that allows the entity to improve all bottlenecks simultaneously"?
Yeah, that's one part of it, the largest one. A second part is that, at any given point, you have a handful of specific bottlenecks where incremental investments produce outsized impacts, so the benefit of full generality is not large. The third part is that improvement that is not "self"-improvement is still improvement.
When I consider all three together it seems really unlikely that there's any particular meaningful "threshold of self-improvement" - the capabilities we actually care about in terms of altering the world in large scale and potentially bad-for-humans ways will probably be unlocked quite a bit earlier than fully-general recursive self-improvement.
There's a mental pattern people (including past me) sometimes have where they basically think of it as there being a "game over" screen that pops up when AI "reaches the RSI threshold", but it sure looks like we've already reached the threshold of meaningful but non-generalized recursive improvement and the situation will continue to look muddy for the foreseeable future.
More options
Context Copy link
More options
Context Copy link
That is indeed enlightening.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link