This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Why wouldn't RSI be possible? Of course it's possible. The most accurate predictors rate it not only possible, but likely, and soon.
That doesn't mean that Anthropic will be "winner" of AI or that shareholders will make any money of course.
Is RSI well defined? I think a lot of confusion in this sub-thread is caused by one side saying that RSI is a model modifying its own weights (which is not happening yet in a way that is meaningful), and the other side saying RSI is researchers using AI to accelerate AI R&D (which is...obviously true. The vast majority of code written at Anthropic and OpenAI is AI generated)
AI capabilities growing linearly or exponentially is RSI. Eventually.
If it is log or nlogn we are fucked but probably not in polynomial time.
The core "worry" about RSI is that it improves in exponential time and thus cascades beyond human understanding or control. Researchers using AI to accelerate AI R&D is mostly "linear" because it requires the researchers to understand and often guide the AI in directions.
Because of this I don't think RSI is linear by definition. Human's centaur-ing is not "recursive" or "self", it's just improvement.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
By what scientific or engineering evidence have you seen of it doing so? Lot of hypers think it's possible and soon but do they have any research to point to it being the case? The theorize it to be the case, but that's a theory on faith not evidence.
Keep in mind that the current "theory" on AI is "Moore's law for AI via Data". That is completely at odds with the RSI theory unless that RSI is creating additional synthetic data that actually beats the sim-data distribution problem.
I'm not sure what you're even asking. You want a research paper that says its possible for AI to develop an AI that is more capable than itself? Are you asking for like an official statement or a mathematical proof or something? You want 100% certainty?
Its self-evident that AI's are becoming more and more capable, tackling longer term tasks without human intervention, and now even solving problems which humans are incapable of. Furthermore, all of this progress falls along a trajectory which has been predicted for years by the same people who also predicted RSI.
RSI is already here. AI's are already heavily involved in the design of AI systems. There are still humans in the loop, but the human contribution gets smaller as a percentage with each generation. All that is missing is the final step removing humans from the design entirely.
The magical thinking is that AI progress will suddenly stop now even though it has substantial runway to advance by scaling alone if nothing else.
Considering we just had a paper on "AI's totally feel pain maaaan" It would take a bit more. But theoretically a demonstrated experiment of some ML/AI specific improvement to LLMs that show an appreciable improvement to the state of the art. That would be strong evidence of self-improvement.
But how? Not by being smarter, but by brute force number crunching or human-AI teaming.
Uhhh aren't we already supposed to be dead by their predications? Or is it that we have 3 more years, or 10 more years? Idk there's been so much sliding around. Along with all these failed predictions about the method to AI improvement. Didn't they bet on GOFAI? It sounds like nobody holds them accountable for their failed predictions which they sweep under the rug.
Outrageous claims need extraordinary evidence. Prove it. Prove that the AI is currently improving without humans, self-recursively.
What is the AI doing in the loop, and what is the human doing in the loop? Loop-percentages are misleading if the human is doing most of the ideation, system design, and out-of-distribution thinking and the AI is doing the more rote or established needs in the project. I have not been impressed by Astra/Sol 6.1 on brainstorming solutions to hard engineering ideas.
I fail to see where I said this.
More options
Context Copy link
That is not at all self-evident. I have seen no such increase in capability, and I still consider LLMs to be poorly suited for the programming world despite the continued hype.
I'm at a big tech company grasping at the frontier.
For the past, I don't know, six months, code review has been mostly a waste of time. Agents look at commit summaries and detect if the stated intent is actually achieved in the code. They check for bugs, style, and anything that a project maintainer might care about that has been shoved into a nearby markdown file. This system is so good now that most of my changes don't actually require another engineer to review them, they can be AI-approved and shipped.
I'm recently on a project where that's not completely the case, but it's the kind of project you'd point to when arguing that actually leetcode problems really are relevant to some jobs. But even then the agents still almost never make a mistake, likely they're doing better than the humans.
More options
Context Copy link
Is your claim that there is not much difference between, say, GPT 6 and GPT 4o in coding capability? Do you reject all benchmarks and the revealed preference of programmers? I could understand (but not agree with) the skepticism a few years ago, but at this point you're clinging to a floating plank and loudly telling people who float by that there is no tsunami.
Yes. I've seen what both can do, and neither is very good, certainly not good enough to actually be a tool which can save me time on my work.
You realize that programmers are being forced to use these tools, right? For those who have a choice, it is very much a minority who reach for LLMs.
Programmers are not being forced to use claude at my company. The programmers started using it first, and it spread to everyone on the team because it worked so well. No one wants to admit to using AI because of the widespread anti-AI activism, but my understanding is that a lot, probably most coders in the game industry are using it, judging by noticeable sudden acceleration in output. Not just coders either; it's not very good at art, but it is very good at a lot of the technical scutwork required for integrating that art.
Do you have some good examples of this? I've been expecting to see it (and looking forward to it), but of the indie/AAA games I play, I wouldn't say they're coming out noticeably faster, more bug-free, or grander in scope. It's possible I'm just unobservant, or maybe (since dev cycles still exist) the real effect will still be delayed by a year or so.
The rate of games being released onto the Steam Store has gone vertical afaik since your basic friendslop content now takes like a week instead of a couple months. Do agree there hasn't seemingly been that big an impact on AAA games but they've got such huge release cycles.
More options
Context Copy link
More options
Context Copy link
Well they are at mine, and they are quietly unhappy about it, but it's been made clear by management that the only acceptable opinions one can hold on AI are positive. So they keep their mouths shut and collect the paycheck, but they hate what the job has become and it hasn't even increased productivity for them. I hear similar stories from friends at other companies.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I would bet large amounts that any opinion poll on this would be overwhelmingly in favour of rapid capability improvement
More options
Context Copy link
I'm creating new apps every couple days from a small local model. The time from idea to implementation is at least an order of magnitude shorter than it was, and the code is more consistent and better documented. It is not (only) hype, it is reality.
Yes. To me it is quite obvious as well.
I went from.
Chat-GPT 4o - Useful as a reference. Can write small scripts and functions.
Chap-GPT 5.5 - Can understand my projects, fix bugs. Still does annoying stuff.
Chap-GPT 6 Astra - I no longer write any code or even check the code that it writes.
The speed has improved by a lot as well.
I use Astra for work stuff, mostly coding and research. But the idea that you don't check it is wild to me. If it fucks up, and it can, it is incredibly professionally embarrassing. People will rightly blame you for the AI-fuckup, they will think less of your competence, and in a research/engineering setting that's professionally bad.
I have a junior that uses Astra for a ton of his work. It keeps tunneling on stuff that doesn't really matter or not really core to the goal of the project. Loses the forest for the trees so to speak. He uncritically just reports what it says and when questioned about why, or what it means he can't explain it. We've extensively talked to him about his AI usage, and honestly if it continues my boss is considering putting him on "PIP" (We don't really pip in the traditional sense).
My observation has been that "good code" looks very much the same for AI as it does for human developers, and that there is a sort of ideal fractal context or documentation where you should be able to start from no knowledge of the codebase, find the relevant sections for your query, and understand the adjacent bits well enough to confidently make changes within a limited context window. Somewhat amusingly, it wants to write too many comments (especially in places that will go out of date), where humans often have left not enough.
Also it's much better at reading code than writing it, IME.
More options
Context Copy link
Yes, this does happen still quite often. There's some common sense element still missing, kind of like an autistic savant.
But code correctness is largely solved. At least, it's going to be more correct than any human.
Sounds like they should just fire him since he's a thin layer between his boss and Astra. Too bad most of us will be in this place before long.
The point of juniors is have them learn how to be their own engineers. AI has made juniors like interns. If you don't invest in them you won't have staffs, seniors, and principles that can act as SMEs to AI's mistakes.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm not convinced, myself. We don't have good measurements of "intelligence", but I won't be surprised if it ends up bounded, at least tractably, sub-linearly in both compute and training data size. Like, I don't think a TV remote ever has a chance of omniscience. If your assumption is "FOOM" as in AI takeover on a very short (months or less) timescale as in Terminator or The Matrix, I have my doubts. If the claim is instead "GDP line go up at faster exponential rate", then that sounds more plausible. The robotics folks seem to be making some serious progress, but actually building an army of robots with human-level dexterity seems like it'll take the next few decades at best.
Us meaty hominids have been doing RSI via "evolution by natural selection" thing for the last million or so years: demonstrably results have been exponential, but not some "goes to infinity at T=0" singularity.
Exponential over a million of years or so, not over a half a decade. Evolution is slow.
More options
Context Copy link
Yes. I think, with low confidence, that FOOM will not happen. I also do not believe that intelligence is necessarily unbounded.
Nevertheless, I think it's unlikely that human intelligence is an upper bound, nor is the sum of all human intelligence. At some point, in the not too distant future, humans will be largely a hindrance in software and mathematical domains.
So far, the skeptics have been wrong consistently.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link