site banner
PaperclipPerfector  3d ago (text post)   12098 thread views

Culture War Roundup for the week of August 11, 2025

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

3
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Training language models to be warm and empathetic makes them less reliable and more sycophantic:

Artificial intelligence (AI) developers are increasingly building language models with warm and empathetic personas that millions of people now use for advice, therapy, and companionship. Here, we show how this creates a significant trade-off: optimizing language models for warmth undermines their reliability, especially when users express vulnerability. We conducted controlled experiments on five language models of varying sizes and architectures, training them to produce warmer, more empathetic responses, then evaluating them on safety-critical tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard benchmarks, revealing systematic risks that current evaluation practices may fail to detect. As human-like AI systems are deployed at an unprecedented scale, our findings indicate a need to rethink how we develop and oversee these systems that are reshaping human relationships and social interaction.

Assuming that the results reported in the paper are accurate and that they do generalize across model architectures with some regularity, it seems to me that there are two stances you can take regarding this phenomenon; you can either view it as an "easy problem" or a "hard problem":

  • The "easy problem" view: This is essentially just an artifact of the specific fine-tuning method that the authors used. It should not be an insurmountable task to come up with a training method that tells the LLM to maximize warmth and empathy, but without sacrificing honesty and rigor. Just tell the LLM to optimize for both and we'll be fine.

  • The "hard problem" view: This phenomenon is perhaps indicative of a more fundamental tradeoff in the design space of possible minds. Perhaps there is something intrinsic to the fact that, as a mind devotes more attention to "humane concerns" and "social reasoning", there tends to be a concomitant sacrifice of attention to matters of effectiveness and pure rigor. This is not to say that there are no minds that successfully optimize for both; only that they are noticeably more uncommon, relative to the total space of all possibilities. If this view is correct, it could be troublesome for alignment research. Beyond mere orthogonality, raw intellect and effectiveness (and most AI boosters want a hypothetical ASI to be highly effective at realizing its concrete visions in the external world) might actually be negatively correlated with empathy.

One HN comment on the paper read as follows:

A few months ago I asked GPT for a prompt to make it more truthful and logical. The prompt it came up with included the clause "never use friendly or encouraging language"

which is quite fascinating!

EDIT: Funny how many topics this fractured off into, seems notable even by TheMotte standards...

Whatever Anthropic does with Claude seems to work. It's the most flavorful model without really trying too hard to be bubbly and quirky like GPT-4o. Of course, it has its own sycophancy issues, but nowhere near as bad as 4o. (The least sycophantic model I know is Kimi K2, which is incredibly cynical, which makes it interesting)

I am more inclined to go with the "easy problem" view, or perhaps a halfway position. Sycophancy isn't an insurmountable problem. If you're not careful, then trying to knock out obvious sycophancy will make the model more prone to looking for ways to subtly achieve the goal of tricking/convincing the user into giving positive feedback.

To a degree, we really must ask ourselves what "warm and empathetic" really means:

  • If a five year old child asks for feedback on an essay, it is arguably almost always true that their writing sucks. That might be true, but it is a tad-bit unhelpful. The most socially adept/instrumentally useful answer (without outright lying) is to praise them for the effort, offer improvements, and tell them to keep at it. Of course, if you're in literary masters course, and the exact same standard of writing is presented before you, some more colorful verbiage might be appropriate.

  • A lot of social interaction is lubricated by white lies, and a lot of what is deemed "politeness" isn't maximally truth-seeking.

Perhaps maximal truth-seeking conflicts with warmth and empathy. It's possible the tails come apart. But I don't think they're outright opposed to each other, and you can probably find a Pareto frontier that makes most people happy.

Perhaps maximal truth-seeking conflicts with warmth and empathy. It's possible the tails come apart. But I don't think they're outright opposed to each other, and you can probably find a Pareto frontier that makes most people happy.

I think the tails will come apart in the marketplace before the come apart on a technical level. LLMs will get enshittified like everything else, if they haven't begun enshittified. They are optimized for engagement and selling access more than they are optimized for productivity. An effective LLM is an LLM that puts itself out of a job in many tasks.

I disagree on empirical grounds. Altman is a snake, but even he agreed that GPT-4o was concerningly sycophantic, and removed it, while framing 5 as less sycophantic. This caused a revolt by giga-fried 4o addicts, and he relented. Of course, the objections were also more general, I was personally annoyed by the sudden deprecation of 4.1 and o3, and the reduced rate limits, which many other people objected to.

Consider this:

Would you pay more for a therapist or a nuclear engineer (presuming you had any use for the latter)? LLM companies are desperately fighting to move up the value chain, they all want to sell their models as equivalent in performance to PhD candidates, or independent agents capable of doing high value knowledge work. That's what brings in the big bucks from other businesses or HNWIs who will pay >$200/m for pro plans. Having a buddy to chat to definitely brings in money, but it's a rounding error in comparison.

They want to make money from both markets, but one just makes way more sense to focus on. Especially since people will prefer intelligent + sycophantic to less intelligent + equal amounts of sycophancy.

LLM companies are desperately fighting to move up the value chain, they all want to sell their models as equivalent in performance to PhD candidates, or independent agents capable of doing high value knowledge work.

I donno man. How much value is there really here? Unless you just let'r rip and see what happens, all those LLMs doing PhD level knowledge work will need to be overseen still by PhD level knowledge workers to check for veracity and hallucinations. It runs into a bit of the "How does a stupid writer depict a smart villain" problem.

And as for the companies that decide let'r rip without adequate oversight, well... I can't venture to guess. Really playing with fire there.

There is probably perfectly adequate shareholder value in getting a billion lonely midwits to pay $10/month rising to $inf/month in the way of all silicon valley service models, and keeping them hooked with the LLM equivalent of tokenized language loot boxes. I'd wager its even the more significant hill to climb for shareholder value.

Fair points, but verification is usually way cheaper than generation. If one actual human PhD can monitor a dozen AI agents, it is plausible that the value prop makes sense.

In a lot of tasks, including AI research and coding, you can also automate the verification process. Does the code compile and pass all tests? Does the new proposed optimizer beat Adam or Muon on a toy run?

There is probably perfectly adequate shareholder value in getting a billion lonely midwits to pay $10/month rising to $inf/month in the way of all silicon valley service models, and keeping them hooked with the LLM equivalent of tokenized language loot boxes. I'd wager its even the more significant hill to climb for shareholder value.

That might be true today (and tomorrow, or next year), but the companies are betting hard on their models being capable of doing much more, and hence getting paying customers willing to shell out more. The true goal is recursive self-improvement, and the belief that this has far more dollars associated with it than even capturing all the money on earth today. Of course, they need market share and ongoing revenue to justify the investments to get there, which is why you can buy in relatively cheap. Competition also keeps them mostly honest, OAI would probably be charging a great deal more or gatekeeping their best if Google or Anthropic weren't around.

Fair points, but verification is usually way cheaper than generation.

Not if P = NP

Fair points, but verification is usually way cheaper than generation. If one actual human PhD can monitor a dozen AI agents, it is plausible that the value prop makes sense.

Not necessarily! It's an adage among programmers that reviewing somebody else's code is often harder than making your own, because you have to figure it all out and then try to create some ideal version in your head and mesh the two together.

It's actually a big issue with vibe-coding - I end up with a codebase I don't understand and then have to do the work of figuring out the framework for myself anyway.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats