This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Dead forum?
Today in AI news, OpenAI has reportedly released GPT-6 Astra. Rumors on X suggests that it scores 99% on Arc-AGI-3, meaning that the benchmark is essentially dead now. For comparison, Fable only scored 30%. Does this mean AGI is here? Many are saying that it is. Color me skeptical, but its getting harder and harder to find concrete tests that AI cannot pass. The god of the gaps gets smaller.
Many intelligenct people have important things to say about this. But what do the 84 year old communists think? Bernie Sanders today has introduced legislation calling for a ban on superintelligence with a 20 year prison term for attempted superintelligence. He also wants (shocked Pikachu) a government regulatory agency. Although he mumbles something about international cooperation, its clear that China will not be subject to these rules so it solves nothing.
I will give him credit for at least recognizing the importance of the issue.
We could be entering a world of unimaginable change. And, given the political systems in place in the US and China, the midwives of that changes will be those on the cusp of senility.
With all the renewed hype around Astra and AI mathematics, I have been feeling that I haven't really seen a response I'm satisfied with from the AI bulls (at least the imminent mass unemployment / AGI / singularity types) to the 80IQ skeptic question of "If AI is so good, why is nothing happening?".
FWIW, if I knew in 2020 that in a few years we would have AI models that can get a perfect score on the IMO for pennies and meaningfully contribute to research mathematics I have expected that we basically live in a sci-fi world as well; yet now we have these models for $20 a month, and literally nothing I'm aware of has happened on a macro level clearly attributable to AI, either quantitatively or qualitatively, apart from the distortion of the AI capex buildout. In fact, it's actually GLP-1's that are basically a complete miracle on a macro-economic and public health level, and where there actually are many macro statistics that are visibly affected by GLP-1's.
On some level there probably is anthropomorphization going on here where the natural human assumptions are that
when in fact the conclusion does not actually follow from the premises; it is possible that AI is extraordinarily economically valuable and powerful for other reasons, but this does not follow from being superhuman at mathematics.
Modern pure mathematics is, as we seem to be finding out, not particularly economically valuable in and of itself; mostly mathematicians are lionized for their intellect because a human with that level of intelligence could obviously do significant amounts of other economically valuable work, but this seems extremely non-obvious in the case of AI.
The "best" response I've seen is something along the lines of "actually AI is still heavily limited by spikiness right now but the exponential is going to keep going bro" which I suppose is honest as to current capabilities and not disprovable, but it does feel like a retreat away from "AI is already crazy good" as many people have been and are touting, and towards magical thinking; somewhere along the line the party line swapped from "AGI imminent" to "AI that creates AGI imminent".
My take-away certainly isn't that AI is a fugazi; obviously it's powerful technology and I'm still quite bullish in the mid-long term, but I've certainly been feeling myself getting anti-hyped with recent releases as a result of this growing discontinuity of any real-world effects.
I’d caveat that there have been some massive changes: translation is basically solved, ‘Rosie-the-robot’ style control is no longer science fiction but a creepy and dumb product, ai-assisted chip development and software engineering is a default option. Recursive self-improvement has been slow, rather than non-existent.
Some of the delay is just logistics. There’s probably over a hundred people worldwide following the AI cancer vaccine dog’s approach, but it’s going to be three or four years before we have clear and convincing academic papers even assuming it works and generalizes, just because that’s how medical trials work. AI driven assistance and funding for nuclear reactors might make building them in the US possible, but it isn’t going to make the NRC respond in less than six months. Silicon fab cycles are measured in years. Structural stuff just can’t be that fast.
Others are tooling. There’s a massive space where many small and medium businesses would benefit from custom code, for example, and AI can do it… but if your interface and sanity checks require a programmer or project manager anyway, it’s not really available yet even if the AI can do it. And Claude Code is not ready for prime time use by normies.
A lot of stuff is also just obscured when it does work. An AI-derived optimization algorithm for matrix multiplication just gets swallowed as a delta in efficiency from expected performance. An AI-built software tool just looks like a (verbose) software tool.
I personally think the effects of current AI on software engineering productivity are somewhat overstated (at my fairly AI-pilled software firm there's been many "AI-assisted" design documents and RFC's of dubious value, lots of rewrites of internal platforms, test suites and frameworks that look productive but haven't really been that useful, and only somewhat modest improvement in terms of actual firm-wide user-legible value), but as a whole I would agree that software engineering productivity has been meaningfully enhanced via AI.
There doesn't yet seem to be meaningful impact on software engineering employment, or anything noticeable on a macro scale as a result of modern LLM coding, but I think I would agree it is too early to determine the wider effects of coding agents one way or another.
I think translation is an great example of what I mean by "if AI is so good, why is nothing happening"?
We've basically had "solved translation for dummies" since 2016 Google Translate NMT, in the sense that the output wasn't great, but someone with no knowledge could bumble through understanding the gist of a brochure or a manual, where previously it might have been literally impossible for them to understand the text without consulting another human. I remember quite a few people in language learning communities I was part of at the time were already getting blackpilled on the full automation of translation as an inevitability all the way back in 2016.
LLM's took that up one step further, since about early 2023 we've had "solved translation for normies" in the sense that LLM's could reliably provide semantically correct translations for pretty much any input/output language pair. Maybe it makes some minor mistakes or it sounds weird to the critical reader, but nothing that really matters in the vast majority of use-cases for the vast majority of people.
Most if not all my friends in 2016 would have agreed that the existence of these models would be the end of human translation, and yet in nearly four years there still has been no significant macro impact on translator employment, or really anything noticeable on a macro scale that might be noticeable if translation was truly "solved".
So then, why is this case? This topic came up when I was chatting with some translators the other day, but I wasn't really satisfied by the points that ended up being brought up.
Is AI translation too expensive? Perhaps plausible for the earliest LLM's, but certainly not any remotely recent model, at least when compared to the fully loaded salary of a first world translator.
Is AI translation not reliable enough? I thought this might be the case, but I was assured that this isn't really true for most translation jobs. It certainly matters to have a human liability shield for regulated medical, diplomatic, legal use cases etc, but most translators don't work in those niches, and for most translation work accuracy is not really that paramount - most people are happy with "good enough" and people who care about translation quality have always been a small minority.
Is diffusion simply incomplete and all the translators will be unemployed in X more months or years? Plausible, but I'm not convinced this can be the entire story either; at this point more than half the planet has tried ChatGPT or Gemini, and AI is top of mind in nearly all corporate boardrooms right now. Even if diffusion is slower than expected, I would think it reasonable that after nearly four years there should be some sort of real world impact.
Are there induced demand effects outweighing the supply increase? Certainly demand for Japanese<->English, Chinese<->English and Chinese<->Japanese translation (and likely lots of other language pairs) has exploded in the last 10-20 years; the expectation now is for games, media, writing to be rapidly localized across many different languages when you used to have to wait years for any kind of translation if at all.
This can't be the entire story though; if translation really was solved, it wouldn't matter how high demand was when AI would be meeting the vast majority of it. Perhaps it's possible that most translation is solved, but induced demand for the translation that is not solved means that most translators still have jobs?
I agree that logistics and tooling are significant bottlenecks for anything that needs to interface with the physical world, but I think it's a logical criticism to make that even solely in the worlds of bits and words there's really been very little happening in the world relative to the capabilities of LLM's.
In general I don't think I've seen a rigorous argument supporting the idea that "AI just needs X more months to develop Y capabilities and it'll have Z real world effect", when it seems that nobody really understands what capabilities are missing from the current models that are currently preventing them from having major real-world impact in the first place.
I'm not sure if I would even go that far. I work at a software company, and I do some consulting on the side. I've been able to watch a lot of AI-coding initiatives from the sidelines due to that consulting, and I see the same pattern a lot.
First, management picks something they can measure. Lines of code, commits, and PRs merged are the most common ones. This causes a certain kind of climber who doesn't actually want to be a programmer to ruthlessly optimize for whatever the metric is. Management lauds that person as a "10x/100x/1000x developer" and they receive glowing performance reviews while spending their days giving presentations to their coworkers about how to avoid being Left Behind In the Permanent Underclass.
At the same time, the rest of the team ends up drowning in this person's "work". Their PRs are 30,000 line, codebase-spanning fever dreams that come with "helpful" documentation that doesn't make any sense. The "spec" is under-defined and the code fails to adhere to it. The tests are made up nonsense. Management softly forces to team to rubber stamp the Rockstar's work. Nobody knows what the hell is going on, defects rise, baffling production down events occur, and nobody can figure out why.
The net productivity for the organization doesn't actually change much. What the technology does do is shift perceived productivity into a small group of people who have offloaded all the work to everybody else.
The con man who job hops has been a rampant problem in this industry for decades. LLMs have just systematized and automated it.
I think if you're skilled and disciplined with your LLM usage, it's pretty unquestionable that software engineering productivity has improved.
For example, I've really enjoyed using Ghostty, Dwarfstar and pi; each of these excellent software projects are written using LLM's with no slop in sight, but importantly these projects are also owned by mitchellh, antirez, and badlogic, who are three incredible programmers in their own right.
That being said, I basically agree with you that ironically, the natural-language programming interface is actually extremely difficult to use correctly, and many AI coding initiatives aren't going particularly well because it's so easy to default into slinging slop and hence productivity improvements are hard to find in many cases.
I'm optimistic that over time the industry will develop better norms and understand how LLM tooling should and should not be used, and we'll all be able to reap the benefits (a man can hope...).
Overall, I am less optimistic than you. It seems like the incentives are too broken for us to reach a new equilibrium any time soon. If we do get there, it'll be on the other side of a graveyard of companies that got too greedy.
Yes, I don't disagree that there will be a lot of churn and pain in the meantime.
I find it's practically a second part-time job to hold back the worst impulses of my non-technical coworkers, the junior developers and management to slop up tickets, documentation and the codebase, which has been somewhat taxing. Unfortunately, it'll probably get worse before it gets better.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
That seems like a big-company problem rather than an LLM problem. I work in a small startup and while I won't claim AI-coding has been without complications, if you tried something like that at my company you would be fired immediately.
That is, the manager would be fired for proving they are totally unable to be trusted with a codebase and have no idea how development works, nor any understanding of what is being done under them.
I've seen it repeat at every scale, from legally-a-small-business to a fairly memorable consult at a Fortune 100 company.
I won't argue as to whether your group has some secret sauce or unusual competency, but I will say that what you have described is absolutely not the norm at the kinds of places that will contract me.
Given your work, is it plausible that you are only going to get contracted by the sort of place that’s absolutely addicted to being Goodhart’d to start with?
I can't rule it out. By the time a company calls me, they're generally fucked to the point where they'll call me.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link