This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
See For Yourself: A Live Demo of LLM capabilities
As someone concerned with AI Safety or the implications of cognitive automation for human employability since well before it's cool, I must admit a sense of vindication from seeing AI dominate online discourse, including on the Motte.
We have a wide-range of views on LLM capabilities (at present) as well as on their future trajectory. Opinions are heterogeneous enough that any attempt at taxonomy will fail to capture individual nuance, but as I see it:
LLM Bulls: Current SOTA LLMs are capable of replacing a sizeable portion of human knowledge work. Near-future models or future architectures promise AGI, then ASI in short order. The world won't know what hit it.
LLM moderates: Current SOTA models are useful, but incapable of replacing even mid-level devs without negative repercussions on work quality or code performance/viability. They do not fully substitute for the labor of the average professional programmer in the West. This may or may not achieve in the near future. AGI is uncertain, ASI is less likely.
LLM skeptics: Current SOTA models are grossly overhyped. They are grossly incompetent at the majority of programming tasks and shouldn't be used for anything more than boilerplate, if that. AGI is unlikely in the near-term, ASI is a pipedream.
Gary Marcus, the dearly departed Hlynka. Opinions not worth discussing.
Then there's the question of whether LLMs or recognizable derivatives are capable of becoming AGI/ASI, or if we need to make significant discoveries in terms of new architectures and/or training pipelines (new paradigms). Fortunately, that isn't relevant right now.
Alternatively, according to Claude:
The Displacement Imminent camp thinks current models already threaten mid-level knowledge work, and the curve is steep enough that AGI is a near-term planning assumption, not a thought experiment.
The Instrumental Optimist thinks current models are genuinely useful in a supervised workflow, trajectory is positive but uncertain, AGI is possible but not imminent. This is probably the modal position among working engineers who actually use these tools.
The Tool Not Agent camp thinks current models are genuinely useful as sophisticated autocomplete or search, but the "agent" framing is mostly hype — they fail badly without tight human scaffolding, and trajectory is uncertain enough that AGI is not worth pricing in.
The Stochastic Parrot camp (your skeptics, minus the pejorative) thinks the capabilities are brittle, benchmark gaming is rampant, and real-world coding performance is far below reported evals. They're often specifically focused on the unsupervised case and the question of whether the outputs are actually understood vs. pattern-matched.
The dimension you might also want to add explicitly is who bears the cost of the failure modes — because a lot of the disagreement between practitioners isn't about raw capability but about whether the errors are cheap (easily caught, low stakes) or expensive (subtle, compounding, hard to audit). Someone who works on safety-critical systems has a very different prior than someone shipping web apps.
Coding ability is more of a vector than it is a scalar. Using a breakdown helpfully provided by ChatGPT 5.2 Thinking:
Most arguments are really about which of these capabilities you think models have:
Local code generation (Boilerplate, idioms, small functions, straightforward CRUD, framework glue.)
Code understanding in situ (Reading unfamiliar code, tracing control flow, handling large repos, respecting existing patterns.)
Debugging and diagnosis (Finding root cause, interpreting logs, stepping through runtime behavior, reproducing bugs. Refactoring and maintenance)
Changing code without breaking invariants, reducing complexity, untangling legacy.
System design and requirements translation (Turning vague specs into robust design, choosing tradeoffs, anticipating failure modes.)
Operational competence (Tests, CI, tooling, dependency management, security posture, deploy and rollback, observability.)
Two people can both say “LLMs are great at coding” and mean (1) only vs (1)+(2)+(6) vs “end-to-end ticket closure.”
With terminology hopefully clarified, I come to the actual proposal:
@strappingfrequent (one of the many Mottizens I am reasonably well-acquainted with off-platform), has very generously offered:
A sizeable amount of tokens from his very expensive Claude Max plan ($200 a month!) and access to the latest Claude Opus.
His experience using agent frameworks and orchestration. I can personally attest that he was doing this well before it was cool, I recall seeing detailed experimentation as early as GPT-4.
His time in personally setting up experiments/tests, as well as overseeing their progress, while potentially interacting with an audience over a livestream.
He works as a professional programmer, and has told me that he has been consistently impressed by the capabilities of AI coding agents. They've served his needs well.
Here's his description of his skills and experience:
To what end?
He and I share a dissatisfaction with AI discourse that substitutes confident assertion for empirical investigation, and we think the most useful contribution we can make is to show the tools actually working on tasks that skeptics consider beyond their reach.
What do we want from you?
If you self-identify as someone who is either on the fence about LLMs, or strongly skeptical that they're useful for anything: share a coding challenge that you think they're presently incapable of doing, or doing well.
An ideal candidate is a proposal that you think is beyond the abilities of any LLM, while not being so difficult that we think they'd be entirely intractable. Neither of us claim that we can solve Fermat's Last Problem (or that Claude can solve it for us).
Other requirements:
A clear problem specification, or a willingness to submit a vaguer one and then approve a tighter version as created by us/Claude.
Nothing so easy/trivial that a quick Google shows that someone's already done it. If you want a C++ compiler written by an LLM, well, there's one out there (though that is the opposite of trivial).
Nothing too hard. He provides an example of "coding a Netflix clone in 4 hours".
An agreement on the degree of human intervention allowed. Can we prompt the model if it gets stuck? Help it in other ways? Do you want to add something to the scope later? (Strongly inadvisable). Note that if you expect literally zero human intervention, SF isn't game. He says: "I don't think I'd care to demonstrate any sort of zero-shot capacity... that's a silly expectation. If my prime orchestrator spends 30 minutes building a full-stack webapp that doesn't work I'll say
It doesn't work; troubleshoot, please. I trust your judgement."A time-horizon. Even a Max plan has its limits, we can't be expected to start a task that'll take days to complete.
Some kind of semi-objective rubric for grading the outcome, if it isn't immediately obvious. Is it enough to succeed at all? Or do you want code that even Torvalds can't critique? And no, "I know it when I see it" isn't really good enough, for obvious reasons. Ideally, give us an idea of the tests everything needs to pass.
If your task requires the model to review/extend proprietary code, that's not off the table entirely, but it's up to you to make sure we can access it. Either send us a copy or point us at a repo.
Nothing illegal.
But to sum, up, we want a task that we agree is probably feasible for an LLM, and where success will change your mind significantly. By which I mean: "If it succeeds at X, I will revise my estimate of LLM capability from Y to Z." Otherwise I can only imagine a lot of post-hoc goalpost movement, "okay but it still needed 3 prompts" or "the code works but a good programmer would have done it differently."
We reserve the right to choose which proposals we attempt, partly because some will be more interesting than others, and partly because we have finite tokens and finite time.
Miscellaneous concerns:
Why Claude Opus 4.6?
Well, the most honest answer is that @strappingfrequent already has an Claude Max plan, and is familiar with its capabilities. The other SOTA competitors include Gemini 3.1 Pro and GPT 5.3 Codex, which are nominally superior on a few benchmarks, but a very large fraction of programmers insist that Claude is still the best for general programming use cases. We don't think this choice matters that much, and the models are in fact roughly interchangeable while being noticeably superior to anything released before very late 2025.
Why bother at all?
We are hoping to change minds. This is disclosed upfront. We're also open to changing our own minds. We would also be genuinely interested to discover cases where we expect success and get failure - mapping the capability frontier is valuable independent of which direction the evidence points. We will share logs and a final repo either way. An interactive livestream is possible if there is sufficient interest.
Anything else?
You know the model. We'll be using Claude Code. The specifics of the demo are TBD, it could be a livestream with user engagement if there's sufficient interest, otherwise we can dump logs and share a final repo.
The floor is open. What do you think Claude cannot do?
Edit:
I'll forward the proposals to @strappingfrequent, assuming he doesn't show up in the thread himself. I am clearly not the real expert here, plus he's doing all of the heavy lifting. Expect at least a day or two of back and forth before we come to a consensus (but if he agrees to something, then assume I won't object, but not vice versa unless specifically noted), and that includes conversations within this thread to narrow down the scope and make things as rigorous as possible while adhering to the restrictions we've mentioned. At some point, we'll announce winners and get to scheduling.
Inspired by @self_made_human 's suggestion, I want to offer a verifiable challenge to create a novel. It's not strictly coding but if you're willing to accept the challenge I think it will be interesting.
The challenge:
Write a 30,000-50,000 word novella with coherent characters, as well as a twist/reveal sometime after the midpoint. I'm purposely leaving the topic open, but happy to make the challenge more specific if that helps. It could be a thriller like by Michael Crichton or something even more ambiguous like John Steinbeck. Verification of the challenge will be done with LLM judges. Any agentic system or techniques are allowed, except for direct access to the judging criteria or plagiarism.
Requirements:
Verification:
The verification prompts will be run using a frontier LLM with a long context window, enough to put the entire novel in the context. The outputs of the verification prompts may be consumed by humans, but if the outcome, pass/fail is ambiguous, the verification prompts themselves should be tweaked asking for a clearer response, and run again. The verification prompts should be run using the API, not the web UI, using the default recommended settings of temperature and other sampling parameters, and run 5 (or more) times each to ensure an accurate result.
In order to prevent an AI agent from "gaming" the challenge, the agent must not be given access to run LLM judges directly on the success criteria. It may also not access the success criteria directly, but may be given it implicitly if phrased as general requests for good writing.
Uh... I dunno that you need a cutting-edge model for that. I used a similar approach for this (cw: bad Jupiter Ascending fan-script). It's not good -- I'd say not even good as fanfiction -- and it's not even what I'd want written for the setting, and it's admittedly only into 13k words. But while it took three layers of "let's take these characters and flesh them out", "let's add this setting flesh out into a story outline", and then finally prompting the actual story, it did do it with minimal human intervention and none of it actually drawing the story plot. Putting even trivial effort into feedback, guidance, and pacing during the final prompting sequence would probably have helped a ton.
My problems are more than the character voices are really samey, the setting doesn't get enough interesting exploration, the twist doesn't get enough emphasis (and frankly isn't that interesting even in outline form: "why would anyone be willing to risk eternity for an unproven chance? Well, we happen to have a big pile of people that risked their lives and were trying to kill for a tiny improvement. Having eternal life only available to the elite kinda makes that a day-to-day thing."), and it keeps throwing extra characters in with too much detail rather than using the ones I was trying to emphasize. It's not necessarily incoherent, just bad.
((The LLMs do eventually notice that it's a Jupiter Ascending-with-names-filed-off-story if you try your review. Not sure whether that hurts or helps it as analysis, but given that the character tones sound nothing like their film counterparts I don't think it pollutes too much. And while my original fic efforts have been on content that you... probably will find even less appealing to read, original fic does work.))
I've got a busy week, but I might see what I can get out of a local LLM aiming for the longer form 30k words target, just to do a compare and contrast.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link