site banner
PaperclipPerfector  3d ago (text post)   12124 thread views

Culture War Roundup for the week of August 11, 2025

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

3
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Training language models to be warm and empathetic makes them less reliable and more sycophantic:

Artificial intelligence (AI) developers are increasingly building language models with warm and empathetic personas that millions of people now use for advice, therapy, and companionship. Here, we show how this creates a significant trade-off: optimizing language models for warmth undermines their reliability, especially when users express vulnerability. We conducted controlled experiments on five language models of varying sizes and architectures, training them to produce warmer, more empathetic responses, then evaluating them on safety-critical tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard benchmarks, revealing systematic risks that current evaluation practices may fail to detect. As human-like AI systems are deployed at an unprecedented scale, our findings indicate a need to rethink how we develop and oversee these systems that are reshaping human relationships and social interaction.

Assuming that the results reported in the paper are accurate and that they do generalize across model architectures with some regularity, it seems to me that there are two stances you can take regarding this phenomenon; you can either view it as an "easy problem" or a "hard problem":

  • The "easy problem" view: This is essentially just an artifact of the specific fine-tuning method that the authors used. It should not be an insurmountable task to come up with a training method that tells the LLM to maximize warmth and empathy, but without sacrificing honesty and rigor. Just tell the LLM to optimize for both and we'll be fine.

  • The "hard problem" view: This phenomenon is perhaps indicative of a more fundamental tradeoff in the design space of possible minds. Perhaps there is something intrinsic to the fact that, as a mind devotes more attention to "humane concerns" and "social reasoning", there tends to be a concomitant sacrifice of attention to matters of effectiveness and pure rigor. This is not to say that there are no minds that successfully optimize for both; only that they are noticeably more uncommon, relative to the total space of all possibilities. If this view is correct, it could be troublesome for alignment research. Beyond mere orthogonality, raw intellect and effectiveness (and most AI boosters want a hypothetical ASI to be highly effective at realizing its concrete visions in the external world) might actually be negatively correlated with empathy.

One HN comment on the paper read as follows:

A few months ago I asked GPT for a prompt to make it more truthful and logical. The prompt it came up with included the clause "never use friendly or encouraging language"

which is quite fascinating!

EDIT: Funny how many topics this fractured off into, seems notable even by TheMotte standards...

Whatever Anthropic does with Claude seems to work. It's the most flavorful model without really trying too hard to be bubbly and quirky like GPT-4o. Of course, it has its own sycophancy issues, but nowhere near as bad as 4o. (The least sycophantic model I know is Kimi K2, which is incredibly cynical, which makes it interesting)

I am more inclined to go with the "easy problem" view, or perhaps a halfway position. Sycophancy isn't an insurmountable problem. If you're not careful, then trying to knock out obvious sycophancy will make the model more prone to looking for ways to subtly achieve the goal of tricking/convincing the user into giving positive feedback.

To a degree, we really must ask ourselves what "warm and empathetic" really means:

  • If a five year old child asks for feedback on an essay, it is arguably almost always true that their writing sucks. That might be true, but it is a tad-bit unhelpful. The most socially adept/instrumentally useful answer (without outright lying) is to praise them for the effort, offer improvements, and tell them to keep at it. Of course, if you're in literary masters course, and the exact same standard of writing is presented before you, some more colorful verbiage might be appropriate.

  • A lot of social interaction is lubricated by white lies, and a lot of what is deemed "politeness" isn't maximally truth-seeking.

Perhaps maximal truth-seeking conflicts with warmth and empathy. It's possible the tails come apart. But I don't think they're outright opposed to each other, and you can probably find a Pareto frontier that makes most people happy.

Perhaps maximal truth-seeking conflicts with warmth and empathy. It's possible the tails come apart. But I don't think they're outright opposed to each other, and you can probably find a Pareto frontier that makes most people happy.

I think the tails will come apart in the marketplace before the come apart on a technical level. LLMs will get enshittified like everything else, if they haven't begun enshittified. They are optimized for engagement and selling access more than they are optimized for productivity. An effective LLM is an LLM that puts itself out of a job in many tasks.

I disagree on empirical grounds. Altman is a snake, but even he agreed that GPT-4o was concerningly sycophantic, and removed it, while framing 5 as less sycophantic. This caused a revolt by giga-fried 4o addicts, and he relented. Of course, the objections were also more general, I was personally annoyed by the sudden deprecation of 4.1 and o3, and the reduced rate limits, which many other people objected to.

Consider this:

Would you pay more for a therapist or a nuclear engineer (presuming you had any use for the latter)? LLM companies are desperately fighting to move up the value chain, they all want to sell their models as equivalent in performance to PhD candidates, or independent agents capable of doing high value knowledge work. That's what brings in the big bucks from other businesses or HNWIs who will pay >$200/m for pro plans. Having a buddy to chat to definitely brings in money, but it's a rounding error in comparison.

They want to make money from both markets, but one just makes way more sense to focus on. Especially since people will prefer intelligent + sycophantic to less intelligent + equal amounts of sycophancy.

They want to make money from both markets, but one just makes way more sense to focus on. Especially since people will prefer intelligent + sycophantic to less intelligent + equal amounts of sycophancy.

I dont think they actually do. IMO a large problem with most AI companies is they are vanity projects being overseen by bloated, already successful, companies that are looking to find a second revenue stream in the future. But that future is far off and the current revenue streams aren't going anywhere soon, so they can afford to be stupid and make their AI's intentionally stupid to placate their employees who don't want to see an AI outputting things that would offend said employees.

I think it matters what you intend the system to be used for. There’s probably a market for a sycophantic waifu or friend bot. But I don’t want my accountant to act like my best friend. In fact, I’d personally trust business or career advice less if I thought the human or bot giving the advice was trying to be my friend or appear as my friend.

LLM companies are desperately fighting to move up the value chain, they all want to sell their models as equivalent in performance to PhD candidates, or independent agents capable of doing high value knowledge work.

I donno man. How much value is there really here? Unless you just let'r rip and see what happens, all those LLMs doing PhD level knowledge work will need to be overseen still by PhD level knowledge workers to check for veracity and hallucinations. It runs into a bit of the "How does a stupid writer depict a smart villain" problem.

And as for the companies that decide let'r rip without adequate oversight, well... I can't venture to guess. Really playing with fire there.

There is probably perfectly adequate shareholder value in getting a billion lonely midwits to pay $10/month rising to $inf/month in the way of all silicon valley service models, and keeping them hooked with the LLM equivalent of tokenized language loot boxes. I'd wager its even the more significant hill to climb for shareholder value.

Fair points, but verification is usually way cheaper than generation. If one actual human PhD can monitor a dozen AI agents, it is plausible that the value prop makes sense.

In a lot of tasks, including AI research and coding, you can also automate the verification process. Does the code compile and pass all tests? Does the new proposed optimizer beat Adam or Muon on a toy run?

There is probably perfectly adequate shareholder value in getting a billion lonely midwits to pay $10/month rising to $inf/month in the way of all silicon valley service models, and keeping them hooked with the LLM equivalent of tokenized language loot boxes. I'd wager its even the more significant hill to climb for shareholder value.

That might be true today (and tomorrow, or next year), but the companies are betting hard on their models being capable of doing much more, and hence getting paying customers willing to shell out more. The true goal is recursive self-improvement, and the belief that this has far more dollars associated with it than even capturing all the money on earth today. Of course, they need market share and ongoing revenue to justify the investments to get there, which is why you can buy in relatively cheap. Competition also keeps them mostly honest, OAI would probably be charging a great deal more or gatekeeping their best if Google or Anthropic weren't around.

Fair points, but verification is usually way cheaper than generation.

Not if P = NP

Fair points, but verification is usually way cheaper than generation. If one actual human PhD can monitor a dozen AI agents, it is plausible that the value prop makes sense.

Not necessarily! It's an adage among programmers that reviewing somebody else's code is often harder than making your own, because you have to figure it all out and then try to create some ideal version in your head and mesh the two together.

It's actually a big issue with vibe-coding - I end up with a codebase I don't understand and then have to do the work of figuring out the framework for myself anyway.

I would argue that this is a temporary state of affairs. Current AI coding is at the level of an over-caffeinated intern (who is very knowledgeable, but less than practical). Thus, a great deal of oversight is necessary to make sure they aren't shooting themselves in the foot.

But consider the potential SOTA in a year or two, when they're comfortably at par with mid-level coders. A senior SWE is usually happy to delegate to multiple experienced juniors, without worrying too much about the exact implementation details. My impression is that we're not there yet.

https://x.com/METR_Evals/status/1955747420324946037

Even when agents pass on all human-written test cases, we estimate that their implementations would take 20-30 minutes on average to get to a mergeable state—which represents about a third of the total time needed for an experienced developer to complete the tasks.

In other words, a lot (but not all) of the theoretical time savings are eaten up by the need to understand, edit and improve their code. At present.

The total salary of all therapists is surely far higher than the combined salary of all nuclear engineers? I tried to find aggregate employment figures and failed.

Broadly, there are huge amounts of people who are very lonely and unable realistically to fix that. I think the value from providing a real-enough friend to them would be vastly more valuable in both utilitarian and monetary terms than almost anything else. I hope of course to move to an open, almost-free solution.

The total salary of all therapists is surely far higher than the combined salary of all nuclear engineers?

Almost certainly true, and my analogy is imperfect.

In the limit, AI are postulated to be capable of doing {everything humans can do}, physically or mentally.

But AI companies today are heavily compute constrained, they're begging Nvidia to sell them more GPUs, even at ridiculous costs.

That means that they want to extract maximum $/flop. This means that they'd much rather automate high value knowledge work first. AI researchers make hundreds of thousands or even millions/almost billions of USD a year, if you have a model that is as smart as an AI researcher, then you can capture some of that revenue.

Once those extremely high yield targets are out of the way, then you can start creeping down the value chain. The cost of electricity for ChatGPT is less than that hourly fee most therapists charge.

Of course, I must hasten to add that this is an ideal scenario. The models aren't good enough to outright replace the best AI researchers, maybe not even the median or subpar. If the only job they can do is that which demands the intelligence of a therapist, then they'll have to settle for that.

(And of course, the specter of recursive self improvement. Each AI AI researcher or programmer can plausibly speed up the iteration time till an even better researcher or coder. This may or may not be happening today.)

In other words, yhere are competing pressures:

  • Revenue and market share today. Hence free or $20 plans for the masses.

  • A push to sell more expensive plans or access to better models to those willing to pay for them.

  • Severe compute constraints, meaning that optimizing revenue on the margin is important.

You’re right, I wasn’t really thinking about extracting max value from limited compute.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats