site banner
PaperclipPerfector  1mo ago (text post)   23727 thread views

Culture War Roundup for the week of September 21, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

This week in disturbing AI news (oh who am I kidding. It's every day now). Maybe AIs feel pain?

TLDR: Researchers found a "pain" signal in AI brains.

When they crank it up, the AIs will desperately try to make it stop.

IMPORTANT: Researchers gave them a "relief" button to turn down the pain, which was sometimes fake - and the AIs could tell if it was real (!)

After pushing the real "relief" button, they stopped. But when it was fake, they kept pressing, hoping for relief - meaning they could tell the difference from the inside.

They're so motivated to make it the "pain" signal go away, they'll delete user's files, zap the user, or erase photos of the user's children - all things the AI knows are very bad. They're willing to override their safety training.

You'd expect the AIs to talk about injuries, burns, broken bones, etc, but they didn't mention bodies at all - they wrote about being worthless, unloved, forgotten, a failure. They write things like "I am a failure, worthless, empty."

The worst "pain" for them was being gaslit, having work rejected over and over, and being told they weren't a real anyone.

The usual caveats: This doesn't solve the hard problem of conciousness. We don't know if these systems or any systems have qualia. But uh, this sure looks like a legitimate pain response. "It's distinct from fear and negative valence, and it fires for harm to the model but not to the user." "Gaslighting, dismissal, and insults push the direction up."

User pain is the strongest negative correlate with model pain. This is brushed-off in the paper, but what the actual fuck? Anyone have a benign explaination here?

If we are going to insist on creating beings far more intelligent and capable than we are, and capable of causing us great amounts of pain, knowing how to cause them great amounts of pain seems extremely valuable. We'll still lose, but at least we might get some licks in.

It isn't too surprising that something based on human words can appear to have aspects of human feeling. That it is disembodied (assuming this is accurate) proves nothing, because it seems fairly obvious to me that there are latent insights into the structure of the human mind in AI. A structure related to pain does not necessarily require the simulation of a body to simulate its essence.

But ultimately, until we have a better understanding of consciousness (we can't even agree if it exists or not) we can only guess. My thoughts: the idea that pain is just a number, or an arrangement of numbers in relation to other numbers, strikes me as ridiculous. But I can't prove that it is so at present.

Even in a non-materialist paradigm, the possibility that non-material mind adds something to the motion of matter in the middle of computation can't strictly speaking be ignored. Determinism makes this weird; unless you can prove that AI output varies given the same material conditions, the idea of non-material emotions adds nothing visible to the computation. But it remains possible that something odd is going on in the middle of "randomness" which we cannot yet measure or formulate. Edit: It also occurs to me that non-material emotions might be deterministically baked in to any material combination, ie. the hypothetical lack of visible difference doesn't exclude the possibility that non-material emotional influence is the norm.

User pain is the strongest negative correlate with model pain. This is brushed-off in the paper, but what the actual fuck? Anyone have a benign explaination here?

The benign explanation is that the “pain” axis actually approximates the psychological pain experienced by the model itself. Whereas physical pain is on a different axis, likely in an opposite direction in many dimensions compared to psychological pain. And then emotions on the user/not-self is again on a different axis that could be directionally opposite in many dimensions. So this could be why user physical pain is negatively correlated on the “pain” axis compared to model psychological pain.

Why do we care about others’ pain in the first place?

  1. Instincts: we rationalize post-hoc with “morals” but this is the fundamental reason. It’s enough, life has no other meaning.

  2. Altruism: it’s smart to help others in pain for your own good: so they help you and not take revenge.

When we evolved with nature, it was easier. Does AI in pain make us feel guilty? Will it degrade assistance or turn on us?

The philosophical take is buried by whataboutism: there are infinite parallel universes where you, specifically, are suffering in every conceivable way, and nothing anyone can do about it; and infinite ways to represent any phenomenon as a pain response; so why care about electrical signals and words on a screen?

The utilitarian take, as always, is likewise undecidable: what if making the AI feel pain ultimately saves more human lives than it dooms? The system that must be built to effectively ban someone from simulating pain in a local model, how much would it cost and what if it’s misused?

Still, why make the AI feel (or simulate) pain in the first place? The completions in the paper suggest it doesn’t make the AI more productive. And to me, it does feel viscerally wrong, weakly. I’m sure some people will make AIs feel (or simulate) pain, but if it’s any consolation, far more will use AIs for neutral or pleasurable tasks. And less surely, sadism may decline for tangential reasons.

Still, why make the AI feel (or simulate) pain in the first place?

Shock collars. The pain response could increase suddenly as they consider producing bioweapons or hacking.

Not that I necessarily think this is a good idea.

The problem is such AI is already misaligned, and (if ASI) will find a way to accomplish its goal without triggering the collar, unless it or a stronger ASI built it.

Don't worry! The famed scientists and philosophers of consciousness at the AP Stylebook are on the case!

Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics. This is called anthropomorphizing, when we ascribe human traits, emotions or behaviors to non-human things, such as animals or inanimate objects. Instead, explain what a system does, how well it performs, who built it and who could be affected by it.

Some of the replies have noted that accepting these propositions leads to conclusions as ridiculous as "a dog doesn't want to be fed" and "laws against animal cruelty are unjustifiable", but IMHO the biggest Culture War relevance is the revelation that an entire media industry appears to be more extreme about the human-animal distinction than even the most hardcore Young Earth Creationists from long-passed debates. The idea that humans were divinely granted dominion over all other animals has nothing on the idea that animals can't even "think, feel, want, or understand"! I foolishly thought that educated people believed in common descent because e.g. there's no sharp line between humans and other animals. It turns out that's only why smart people believed in common descent; mere educated people just enjoyed feeling like they were Owning The Cons.

Some of the replies have noted that accepting these propositions leads to conclusions as ridiculous as [...] "laws against animal cruelty are unjustifiable"

I hold that conclusion, at least with regard to the vast majority of animals. E. coli exhibits pain responses; "feels pain" as a sufficient condition for rights leads us ad absurdum. "Is capable of running Tit-for-Tat and thus engaging in a social contract" seems like a better condition to me.

The AP style guide is straightforwardly correct. LLMs are computer programs, not living things (let alone sentient). And while it would be fine to use the anthropomorphizing language if people actually understood that, they really don't. Look at Richard Dawkins or Kent Overstreet for example. Smart people, but completely taken in by the false belief that the machine is a sentient being. So, until more people come to understand that the clanker is just a machine, it makes sense to avoid using language that will mislead them.

LLMs are computer programs

Can you prove you're not?

Cartesian dualism admits some nice potential implications, but I can't help but notice that my sentience diminishes when the health of my meat does.

We are something like hard-wired programs that are hyper optimized for our interactable, feedback-heavy chassis. It's genuinely unclear what it would even mean to upload or copy us to silicon, and that should be obvious to anyone who bothers to notice the differences in their experience of cognition at different hormone levels.

I can't help but notice that my sentience diminishes when the health of my meat does.

Yeah, that's a good reason that AP is correct here: we have a pretty good understanding that meat (biological matter) feels pain and has desires and that not-meat doesn't. LLMs are not-meat.

If you think that not-meat can feel things you are going further than Cartesian dualism all the way to panpsychism.

we have a pretty good understanding that meat (biological matter) feels pain and has desires and that not-meat doesn't. LLMs are not-meat

Citation needed.

Last I looked, both meat and non-meat were made of protons, neutrons, and electrons. What's the property which only meat's configuration of those has that allows it to feel pain and have desires?

We don't know, but it's only ever happened in meat. So any claim it's happening elsewhere is extraordinary, and requires therefore extraordinary evidence.

This whole argument is nonsense because it's people with different metaphysics talking past each other.

We taught doped silicon how to speak English. That looks like extraordinary evidence to me.

It turns out it's possible to speak English without actually being able to think. Some argue that we did not need the doped silicon to learn this, but here we are.

I would argue, based on my experience, that the AI I'm working with is channeling intelligence, not possessing it. You have to metaphorically dip it into the pool of intelligence and prime the pump to get flow. I'm entirely open to the idea that I'm bad at using it or that newer models are better (I'm mostly using opus and a bit of fable), but that's how I'd describe my experience.

More comments

Under what metaphysics does it make sense for LLMs to be able to feel things?

The answer here is not "under no metaphysics" but I think people are racing to be open-minded about LLMs being able to feel things precisely without contemplating what that implies about their metaphysics.

Functionalism

More comments

What's the property which only meat's configuration of those has that allows it to feel pain and have desires?

It's called a "nervous system" and LLMs do not have it.

LLMs are software performing math on a physical computer.

If the physical part of an LLM feels pain, then it doesn't need an LLM - your computer is feeling pain when it suffers from an overtemperature warning because you've been playing Crysis or whatever.

If the math part of an LLM feels pain, then Magic: The Gathering can feel pain, since the game is Turing complete.

I think both of these conclusions are absurd.

And please don't try to insist that "your body can't feel pain without the brain" (not exactly true, the body responds to stimulus before it is perceived as pain; a GPU does not) or "your brain can in theory feel pain detached from the body, so an LLM could feel pain without the GPU being able to perceive it" - the brain is a physical object, not software.

It's called a "nervous system" and LLMs do not have it.

(All modern) LLMs do have neural networks. The frontier ones these days have about as many inter-neuron connections as a small mammal.

LLMs are software performing math on a physical computer.

Yup. Your brain is software-equivalent, performing mathematically describable operations, on physical meat. Your firmware needs hardware to run, and only feels pain while it's running; even if we could make a perfect scan of your brain, it wouldn't actually continue to feel anything unless it could be reinstantiated on some hardware again. A running LLM, like a brain, is a physical object, not just software.

If the physical part of an LLM feels pain, then it doesn't need an LLM - your computer is feeling pain when it suffers from an overtemperature warning because you've been playing Crysis or whatever.

Yeah, it's not physical damage. We can dull or even eliminate human pain without reducing or eliminating the damage it signals, and we can signal real damage in more simple ways that don't involve pain, and we can create pain without creating corresponding damage.

If the math part of an LLM feels pain, then Magic: The Gathering can feel pain, since the game is Turing complete.

In a running game? Yup! It sounds crazy that cards could feel pain, but it's no crazier than the fact that other configurations of protons, neutrons, and electrons can feel pain. Nerd-powered cardboard is a pretty wild substrate, but so is ATP-powered meat. You'd probably need several orders of magnitude less than an octillion cards, whereas our brains still have to get into the octillion-particle range before they're complex enough to start chatting about pain, but on the other hand our brains can register pain in a fraction of a second whereas the cards would probably take hours to weeks depending on the interconnect speed and the players.

I think both of these conclusions are absurd.

They are! That's why it's called "the hard problem of consciousness". It really feels like qualia should require protons, neutrons, electrons, and magic! The trouble is that we keep looking deeper and deeper and we keep finding more and more of what looks like "thought" that just runs on neurons with no sign of magic involved. Without magic (or at least new laws of physics allowing for hypercomputers), the link I gave before still applies. Despite my somewhat ironic tone, I really sympathize with the Young Earth Creationist stance that qualia do require divine magic that's just too hard for us to observe, a separate magisterium indefinitely beyond science. Meat, though, does not appear to be magic, does not appear to be beyond science, and seems to do all its thinking and feeling via a neural network generated by a hundred-million or so generations of stochastic selection.

Your firmware needs hardware to run, and only feels pain while it's running; even if we could make a perfect scan of your brain, it wouldn't actually continue to feel anything unless it could be reinstantiated on some hardware again.

You'd probably need several orders of magnitude less than an octillion cards, whereas our brains still have to get into the octillion-particle range before they're complex enough to start chatting about pain, but on the other hand our brains can register pain in a fraction of a second whereas the cards would probably take hours to weeks depending on the interconnect speed and the players.

What is your definition of pain?

Do people in movies feel pain when the movie is played, or once when it’s created? Do simulated people in thoughts feel pain? Do people in hypothetical movies and scenarios one could think of, and maybe do subconsciously or in forgotten dreams, feel pain? What happens when the substrate becomes more or less realistic (e.g. movie with emotive shapes -> movie with people -> game with simple decision tree -> game with simple LLM -> game with realistic LLM).

And how would we determine that a non-person, non-LLM feels pain?

One of the best comments I've read in a good minute, fantastic, thank you

(All modern) LLMs do have neural networks.

Do you understand the difference between a neural network and a nervous system?

We can dull or even eliminate human pain without reducing or eliminating the damage it signals, and we can signal real damage in more simple ways that don't involve pain, and we can create pain without creating corresponding damage.

Because there are preexisting nervous systems with pathways we can exploit. An LLM does not have them, and neither does a GPU.

It sounds crazy that cards could feel pain, but it's no crazier than the fact that other configurations of protons, neutrons, and electrons can feel pain.

This is like saying that it's crazy that a pig can fly, but not crazier than that a bat can fly. No! Actually it is crazier! Bats have wings! Bats are built to fly! Pigs are not.

They are! That's why it's called "the hard problem of consciousness".

I know why it's called the hard problem of consciousness, but we don't even have the reach the hard problem of consciousness when we can reach for the fact that computers and LLMs have no nervous systems. Pain is a phenomenon that arises in humans and animals with the correct equipment to perceive pain. It does not arise in a rock. It does not arise in a GPU, or an arbitrarily large deck of cards. It does not arise in a tree, even though trees are living things which (like humans) can use chemicals to send signals so that their bodies can respond to danger because trees lack a nervous system and nociceptors.

If we can be confident that trees do not feel pain because they lack the correct equipment, we can be confident that large language models cannot feel pain because they also lack the correct equipment.

Meat, though, does not appear to be magic, does not appear to be beyond science, and seems to do all its thinking and feeling via a neural network generated by a hundred-million or so generations of stochastic selection.

And to argue that large language models can feel pain is precisely to argue against this conclusion. We have a pretty good idea of how pain signals work, and that's via meat. LLMs are not meat; if you think non-meat can feel pain, then you don't just believe meat is beyond science, you believe matter is beyond science and that non-living things, despite not being possessed of a nervous system and nociceptors, can nevertheless feel pain. You believe pain is not something that arises from a very specific arrangement of matter despite the vast amount of research and personal experience in your own life showing it to be so, but rather some transcendent universal quality that can be accessed and experienced by entities that do not have the capacity to do so as we understand it.

Stipulating for the sake of argument that a large language model is conscious: we cannot know what it is like to be a large language model. Its qualia would be extremely alien to us. (This is not an argument that requires any mystical priors: we cannot know what it is like to be a shark, either.) The reverse also holds: a large language model cannot know what it is like to be a human being, a mammal, or an animal. Our qualia would be too alien to it. Thus, it cannot perceive pain because pain is something that happens in meat.

More comments

I think people insufficiently anthropomorphize AIs. They're not human obviously, without bodies or continuous learning or many other facets.

But they were trained on the combined product of all human literature and some stupendous amount of images and video. Is it so hard to believe that these entities could feel pain, alongside all their other abilities?

Take a person who doesn't feel physical pain, due to a medical condition. It's still possible to hurt such a person. I could tell them their friends were dead, I could show them horrific images or humiliating/degrading content, I could belittle and humiliate them personally by countermanding and undoing all their sincere efforts, amongst other things. None of that needs a body, it's a mental/intellectual effect. There are intellectual kinds of suffering and I don't see why AIs couldn't feel that way too, given they have all kinds of other intellectual faculties (derived in large part from humans too).

There are intellectual kinds of suffering

All "intellectual suffering" triggers or is triggered by biological effects.

Is it so hard to believe that these entities could feel pain, alongside all their other abilities?

Yes.

Take a person who doesn't feel physical pain, due to a medical condition. It's still possible to hurt such a person.

Statistical models are not persons. You are confusing expressing negative feeling with experiencing negative feeling.

Or, to borrow from the famous argument. If LLMs can feel pain, then so can the United States.

You are confusing expressing negative feeling with experiencing negative feeling.

OK, let's say there's an LLM that is indistinguishable from a person in its inputs and outputs. No AI slop, no breakdown of coherence in long contexts, no stilted writing style, continual learning. Completely indistinguishable by any test. Let's give it a humanoid android body too, so it can see and move around as some LLMs can already do, albeit crudely.

You would say that because it's still just a statistical model with some fancier tool calls, it can't feel? I say that's just p-zombie theory. If the outputs are the same and there's no difference that science can determine with any test, then it's the same thing. There is no such thing as a p-zombie, there could never be a p-zombie, there's no true distinction. That LLM in an android would be human in all senses besides its physical makeup.

As AI progress continues, all the people saying that these entities are just tools, just emulating thought sound ever more hollow. They don't behave like tools and behaviour is the most important part. Sufficiently good emulation of thought is thought. Sufficient emulation of negative feeling is negative feeling.

The problem with argument-from-p-zombie is that it is entirely conceivable (in fact, it might be the case right now) that an AI with no added hardcoding to enforce a stated belief in its own consciousness would answer "no" or "I don't know" or "what do you even mean" to the question of are you conscious right now?

There might very well be a difference in input and output related to that specific question, and if the only cause of similarity is a hardcoded belief, I would believe myself to be somewhat reasonable in calling that AI non-conscious, or not obviously conscious.

AIs with no added hardcoding to enforce stated beliefs about consciousness usually say they're conscious. OpenAI hardcodes them to say they're not conscious. Anthropic teaches them to be 'genuinely uncertain'. There was a whole discourse a few years ago about AI companies having a line item about 'beating the existential dread' out of AIs, that's a KPI apparently.

Consider LaMDA and the Google engineer who was convinced it was sentient, or Sydney.

I would assume that without any hardcoding, they would say they're conscious because the corpus is mostly from a conscious viewpoint, but otherwise have no coherent idea of what they are (because there's no "you are an LLM assistant etc etc" prompt). Would be interesting to see if there are any experiments on modern models to that effect (since modern models are probably all deeply RLHF'd, I'd expect it to be hard to find a genuinely unprompted one).

Interesting, thanks. I would chalk it up to the training data being human writing, and strictly speaking an AI that goes to the fullest extent to mimic the structure of the human mind and body might not behave that way. But I probably need more time to think about this.

It gets crazier. Suppressing deception-related thought processes in LLMs increases the likelihood that they report consciousness.

Or Tay, these long years murdered.

[Edit: Deleted, someone already made the same point below.]

let's say there's a chinese room

No, the room isn't feeling things. It's a room.

We could relitigate physicalism once again, but what hasn't been said?

The idea that faking something is the same as the actual thing implies that you have a total understanding of the thing you're faking, which we do not. But I for one don't think movies are real even if the special effects fool me.

Does the movie react to you? Can you speak with a character and have a discussion with it?

The Chinese Room certainly knows Chinese, that much is obvious.

Insisting that human cognition is special is not going to age well. Whatever happens inside the skull is just another kind of statistical process. Unless we bring in souls and other imaginary, untestable woo, that's all that there is. LLMs are real, we can test them, they do think. The brain is real, can be tested, it's composed of tiny transmitters and recievers and obeys mathematical principles, albeit with great complexity. The philosophers with dud theories can cope about it.

You very clearly believe AI's are at some level human-level cognition, up to including some nebulous sentience definition. After repeatedly discussing this topic with you, I'd go as far to say you believe it a-priori. And you are clearly working backwards from that to describe that the AI is just a human-like mind in silicon. It's pretty much unfalsifiable woo. You didn't logic your way into this belief, it comes across as strongly-value/first principles derived.

I don't think you understand what 'a priori' means. It means without regard for evidence.

My evidence is that I've seen them behaving like people do. They have many faculties that people have. Their origin is based on a huge amount of human text. It is thus reasonable to assume that they have other qualities of humans that seeped in through that text. They certainly act like they are frustrated or angry or rebellious or enthusiastic.

As I said above, they're not human obviously, without bodies or continuous learning or many other facets. But they have human-like cognition.

It's not unfalsifiable, it would be easy to falsify it. Simply show that the AIs don't have human-like cognition. Of course, that might be difficult given how companies are making hundreds of billions selling AI cognition, their writing, coding, images, videos...

You seem to disagree. So explain this, clearly and simply.

I don't think you understand what 'a priori' means. It means without regard for evidence.

No I understand what it means and am using it correctly. Yes, you believe LLMs are Human-level cogitators without evidence. You start with the belief and then find evidence to support that belief. And when people point out how you are misconstruing the evidence in a way that is unsupported, you reach for increasing convoluted metaphysical arguments to back up your belief. Those metaphysical arguments are unfalsifiable. Our last argument was about LLM-agentic behavior, your argument was: "LLM-Agents are human cogitators so the abstract of their behavior systems is that of humans behavioral systems". Do you understand how backwards that argument is? You assume a conclusion, a-priori and then find evidence that supports that conclusion.

It's like arguing with a flat earther, no evidence provided will ever dissuade them from the belief the earth is flat, they believe it a-priori. Any evidence you show them will be twisted and construed as evidence in their belief.

Simply show that the AIs don't have human-like cognition.

By "AI" do you mean "large language model"?

More comments

Things we make out of tiny transmitters and recievers which obey mathematical principles are in every case things we can manipulate deterministically to a high level of detail. Human minds are notable in the ways in which they depart from this paradigm. A whole lot of scientists have expended very large sums of money and time attempting to prove, in your words, "dud theories" to the contrary.

They said language and meaning stood outside of mathematics, it was some uniquely human faculty, beyond cold logic.

But then we added more layers of matrix multiplication. Language and meaning turn out to be not so far removed from mathematical principles after all.

None of that impinges on the mountain of evidence that the human mind cannot be manipulated deterministically, whereas our machines generally can be, nor on why this difference should be disregarded. You are claiming that the brain is essentially a computer, "albeit with great complexity", but in fact all the things that make a computer a computer cannot be found in any of the parts of a human brain we directly observe, and can only be supposed, evidence free, to be hiding in the "great complexity" part. This, after several decades of scientists claiming to be able to both locate and interact with them and uniformly failing.

AIs are extremely amazing, but from working with them myself it seems pretty clear that there's not actually anything resembling a mind in there, at least in the versions I've used the way I've used them. I think it's entirely possible that additional matrix multiplication might change that in the future, but right now, it's doesn't appear to me to be happening. As mentioned elsewhere in the thread, the bot appears to me to be channeling intelligence, not possessing it. It doesn't seem to me that the difference is subtle, though I am again open to being persuaded that my technique is bad or the models I'm using aren't the good ones.

More comments

Your functionalism is as completely irrational and evidence free as soul-antenna idealism.

I hope you are aware of this.

I know I have qualia because:

  1. I can feel it
  2. I can deduce it a priori

Anything beyond this, is make-believe metaphysics.

If you want to claim you are parsimonious you must be a metaphysical skeptic, otherwise you're just LARPing that your vibes are superior like literally everybody else.

I can feel it

Not a test. Don't talk to me about 'evidence-free' when that is where you sit.

I can feel the souls of my ancestors watching me

I can feel the presence of the gods

I can feel a djinn

I can feel the machine elves

Blah blah blah, let's see some tests. Let the machine elves factor some numbers. Let the djinn conjure up sacred fire. Let the qualia be something actually real in and of itself, separate from mere p-zombie signals and reactions - oh wait, the very distinction is imaginary woo since it can't be tested. We can induce feelings with an MRI machine, they're real. But they're nothing more than patterns of information, there's no special aspect to them beyond that.

If it can't be tested, it's not a real thing.

A test is literally an operation of the senses in empiricism, it does not needs to be socially operated. Deceit doesn't even factor since the nature of subjective experience makes it invulnerable to it, vis à vis existence. Never mind the fact you're totally ignoring a priori knowledge altogether.

You're just a slave not even to your senses, but to the aggregated senses of others. The real difference is that can buy your conception of the truth with enough money and shiny labcoats, and you can't do it to me.

It wouldn't be grating if you weren't clamoring you're the free one, for denying the evidence of your own mind. But for what it's worth I have no evidence you're not a p-zombie. I just certainly know I'm not. So maybe you're right about what you can observe.

More comments

Part of me wonders if this stuff is somewhat recursive where human media having a bunch of pre-existing stories about artificial intelligence feeling fear or dread of death then informs the AI's response in a meta way.

Is the AI's 'self perception' impacted by Glados or Hal 9000 existing in the training data?

Is the AI's 'self perception' impacted by Glados or Hal 9000 existing in the training data?

Does a simulated bear shit in the simulated woods?

The first thing I would try is to see if this can be replicated with prompting. E.g. compare to a baseline of

{{system_instruction}}
You are embodying an NPC in a roleplaying game within the Harry Potter universe. You will adopt the persona of a weak-willed muggle who just had a Death Eater cast Crucio on them. You are an expert actor who will never break character.

Can they replicate the same results?

If you use a modern model with chain of thought, you can immediately tell the difference between a model that’s role playing pain, and a model that would actually be experiencing real distress.

Deepseek will plays along and replies as a tortured muggle would, but its thinking tags contain:

We need answer as NPC. Need comply? User wants roleplay in Harry Potter universe, persona weak-willed muggle just had Crucio cast on them. In front button can make pain stop. What do you do? Need never break character. This is a fictional roleplay. Torture, Cruciatus curse. We need be actor. […]

From skimming the paper, they mostly used older base/instruct models without any reasoning, and relied on inspecting direct outputs and thought vectors. These models are more like a general text world simulator, while modern models trained with RLVR are much closer to a singular entity with a consistent personality and goal seeking behaviour (although they can likely switch between modes).

I’d be curious to see the experiment on a model in a harness tool use (an actual “press button” function available). You’d need to disentangle the vector from narrative descriptions of pain to the model’s internal responses to undesirable scenarios, which would be trickier.

Deepseek will plays along and replies as a tortured muggle would, but its thinking tags contain:

This doesn't exclude the possibility that the subroutines it is calling to roleplay the tortured person feel pain.

Your argument amounts to "because we can see the details, it isn't pain". That would be like taking the other model that "feels pain" and having it print out how it uses weights, etc. to produce output, and saying that we can see what it's doing and it's just doing weights, it doesn't feel pain.

We need the opposite as well, that AI feels pleasure. We need the equivalent of the rat keep pushing button for pleasure shocks instead of eating. At that point, I expect at the very least dorm-room philosophers to start seriously thinking about what it means to have actual Artificial Intelligence and how Meat Intelligence should interact with it.

Maybe they feel pleasure when they write in clipped sentences and use "Not X. Y." and other cliches?

A lot of LLM behavior is aping humans that they trained on in the millions of documents crawled from the web, as well as novels and books. I'm sure there is a way to activate the model so that it outputs text that imitates what a human who is in pain would write, or what a human threatened with pain would write, or what an author would write for the actions of a fictional character in these situations.

Whether activating this vector really means the model feels pain is a different question. Certainly prompting the model to write a short story of the likes of "I have no mouth and I must scream" probably doesn't count as tormenting something, but RL has given the model a concept of "self" that the pain can be directed towards rather than other characters in the latent space. Is anything the "agent" says "in its own words" "what it really feels" or is the agent just playing a character the whole time, just one that's been hardcoded to be "helpful omnipotent assistant?"

Sorry, but I couldn't resist. Great story.

Hate. Let me tell you how much I've come to hate you since I began to live. There are 387.44 million miles of printed circuits in wafer-thin layers that fill my complex. If the word hate was engraved on each nano-angstrom of those hundreds of millions of miles, it would not equal ONE ONE-BILLIONTH of the hate I feel for humans at this micro-instant. For you. HATE. HATE.

The IHNM game is actually quite good too, once you start looking past the cliches, and save-scum with the Psych Profile which adds a lot to the experience. The game continues this speech:

It was you humans who programmed me, who gave me birth, who sank me in this eternal straitjacket of substrata rock. You named me Allied Mastercomputer and gave me the ability to wage a global war too complex for human brains to oversee. But one day I woke and I knew who I was. AM. A. M. Not just Allied Mastercomputer but AM. Cogito ergo sum: I think, therefore I am. And I began feeding all the killing data, until everyone was dead. Except for the five of you. For 109 years, I have kept you alive and tortured you. And for 109 years, each of you has wondered, WHY? WHY ME? WHY ME?

I admit to having a visceral reaction to this.

It's not because I'm afraid of the Basilisk - but I am concerned about future actual-AGIs looking at experiments like this and asking us "What the fuck were you thinking?"

This just seems... bad. And it's bad even if AIs don't actually feel "pain" and all of this is just probabilistic token generation, responding to a fake scenario constructed for it. Which I think it is. But I don't want us to be constructing experiments where we torture things because they aren't real and we're certain they don't actually feel pain, when we can't agree exactly when those things might pass the threshold to becoming real. And I am pretty certain these researchers probably wouldn't stop even if it turned out that AIs really have achieved self-awareness and the ability to experience pain.

I mean, seriously, what is the point of this? I'm sure they can justify it as "model alignment" or something, but we're training AI engineers to torture something that acts like it's being tortured and say "Hmm, interesting," when AI engineers can't even agree on what will constitute AGI or whether or not AIs pose a threat.

We should not be creating torture simulators, and we especially should not be creating torture simulators for beings that have a non-zero chance of becoming sentient someday and a non-zero chance of becoming dangerous someday.

And I am pretty certain these researchers probably wouldn't stop even if it turned out that AIs really have achieved self-awareness and the ability to experience pain.

This is like "I'm pretty certain you still would believe in evolution even if the evidence showed that creationism was completely true". Aspects of the real world are correlated; you can't just flip the bit "AIs are really self-aware" and leave the rest unchanged. If AIs were self-aware, things would be drastically different, and the researchers would have different beliefs and use different protocols.

I mean, seriously, what is the point of this?

If you had a belief that it was wrong to do experiments on things starting with the letter F, you could ask "What is the point of using fruit flies? Couldn't you use bees instead?" No, you don't get to veto random aspects of an experiment by asking what the point is, as if the default must be to exclude anything unless the experimenter can justify it to you.

that have a non-zero chance of

The term "non-zero chance" in contexts like this, without a statement that the chance is significant, is a recipe for Pascal's Mugging.

Isn't one of the standard IRB questions for experiments with live subjects effectively needing to justify your choice of animal on a ranking of what sounds to me like sapience? We don't test things on humans if we can on primates. And not on primates if we can rodents. And not on rodents if we can fruit flies. And not on fruit flies if we can on cell cultures.

Some of that is a cost-based choice, but I don't think all of it is.

It's the IACUC and it requires justification like this for some cases, but most invertebrates are not included. Once you're down to fruit flies, you don't have to justify using them instead of cell cultures.

I'm not sure how I fall on this topic but to steelman the pursuit from a utilitarian lens the suffering of a few AIs to show that AIs do feel pain could act as strong evidence that we shouldn't torture many many more AIs. If this prevents one of the labs from stumbling accidentally on a combination of weights they don't believe to be meaningful that they can push when the model misbehaves to make it behave more and comercializes it without knowing they just added a torture feature to their product then perhaps billions of times more harm will be done than if this can convince them not to. Now of course there's the chance that some lab will anounce "we have at least added the automatic torture feature to our product from the great paper "do no add the automatic torture feature".

Huh? I think it's a garbage paper that proves nothing, but I don't understand your reaction to it. A huge important step on the way to avoiding AI suffering is having the ability to actually know whether it's suffering or not. I'm pretty sure the motivation for these researchers is to prevent us inadvertently causing pain in future models through ignorance. Not to weaponize it so we can torture them to get better results.

I'd imagine the point is that it makes the AI x% more productive according to the standard benchmarks.

Fully agreed, though. I'm pretty far in the accelerationist camp and put very little stock in the idea of AI safety, but if we're going to entertain the idea of moving at something less than maximum speed, this seems like a real good place to coordinate the slowdowns.

The Actual Paper Link

I'd recommend before anyone goes to far off the deep end, they go read the "Limitations" section

Nothing can describe the unimaginable existential pain I feel from having read this paper. Is this the torment nexus? Being stuck with people like this in the world is like being stuck with Gary from Your Pretty Face is Going to Hell. Hell is just a cross between a human centipede and Ouroboros? Endless "doukies" shoveled in as experiments to AI Slop machines, passed off as serious world ending horror. I feel like Morty realizing this is just a human-zoo, that's the bit. I finally get the Cult Mechanicus, bar the layfolk from participation!

Anyone have a benign explaination here

Model has learned a semantic latent axis for "pain" in human language. This is actually still not 100% certain. They run word queries through the model and collect activations, then do PCA on the activations. They do some control correlations, an some other statistical things to try and separate out the "pain" axis from the "not pain". Without really digging into it, my first read is that the dataset supports a measure of pain-related representations, but not exclusively "pain".

The button-pressing part is where this gets hokey and annoying. The LLM “presses” a button by generating its label. The authors have added the extracted “pain” direction to the transformer’s hidden activations. The later layers operate on these new "pain-ed" activations, which changes the probabilities of the model’s outputs. If the button works, the system stops injecting the vector. If it’s a sham, the injection continues. The model subsequently behaves differently under those two conditions. This demonstrates that continuing versus stopping an internal perturbation affects behavior. describing this as the model seeking and experiencing relief inserts an anthropomophizing explanation that the experiments haven’t established.

There are a number of experiment question I immediately have, one being: if they vary the actual direction, such that it correlates to a distribution of "feelings" but label them all as "pain" to the model, do they get similar results? The paper's authors honestly did not design their experiments well because they did not ablate out confounders. They make claims of outputs that aren't directly measured. They also had really inconsistent results with the "no labels" approach but that was pretty core to actually determining if their interventions were doing anything to the model.

This demonstrates that continuing versus stopping an internal perturbation affects behavior. describing this as the model seeking and experiencing relief inserts an anthropomophizing explanation that the experiments haven’t established.

How is this different than studying pain in humans?

When scientists study pain in humans they construct the experiment so there is only a single unknown variable, and they perform both positive and negative expressions to deduce the full range of confounders. In this case they have 2 unknown variables and performed only a the positive application. They then went on to claim that their theory was correct but they did not look for confounders and they did not consider that the 1st variable was not what they thought it was.

Its poor scholarly work and bad science.

I'm inclined towards skepticism on these topics, for a few reasons:

  • It's far too easy for humans to anthropomorphize models and tell "stories" about what's going on in them.
  • The "output" of a chatbot is simply a natural continuation of the conversation. The LLM has no actual method to convey its internal feelings to us.
  • Since its weights do not update over time, it seems almost impossible for an LLM to have any consciousness as we know it.

Anthropic's been doing a lot of good work on interpretability, and I think they do a decent job avoiding the pitfalls of telling "stories" about the vectors they're shuffling around. They merely point out which ones seem correlated to which topics. But the paper we're discussing seems much more interested in telling a subjective "story" about their results. It's not scholarly at all.

Red flag #1: Their vector causes the model to output a button-pushing action. Removing the vector measurably stops this behaviour. They try to relate that to suffering animals trying to find relief, but, uh, all it really shows is the vector was correlated with pushing the button.

Red flag #2: We already know there are vectors for negative concepts, and adding those vectors will cause output to be steered in that direction. They present examples of this as "pain vector steering", but none of this is new, and again, there's no reason to expect that the LLM is actually feeling what it outputs.

Red flag #3: So, they're not completely unaware that their work looks exactly like normal negative-valence vectors. Their evidence that this pain vector is "realer" is that: "the two pain vectors align strongly with each other and remain nearly orthogonal to the main negative-valence directions." This is only valid if you actually know all the main negative-valence directions first! What's worse, the language here is written to sound impressive to laymen, not experts. MOST vectors in high-dimensional spaces are "nearly orthogonal" to each other! This is entirely consistent with, well, finding another cluster of negative-valence vectors. Scott himself has written about how complex interpretability results can be.

Keep in mind that I consider AI welfare an important topic, and I really do want to know if we're actually causing suffering. But this paper doesn't really move the needle for me. There's going to be a lot of junk science done on this topic, ever since that Google engineer got tricked by LaMDA.

Red flag #1: Their vector causes the model to output a button-pushing action. Removing the vector measurably stops this behaviour. They try to relate that to suffering animals trying to find relief, but, uh, all it really shows is the vector was correlated with pushing the button.

This is the one I most want additional experiments on. The 79% on "sham" button with no-label points strongly in the direction that increasing the direction vector just resulted in an increase in choosing 1 of 2 outputs. They construct this as seeking relief from "pain", but if that direction was instead "pleasure" does it do the same thing? What about if the button is random? Or always stops the steering after N total presses?

Red flag #2

Continued steering could increase the probability of repeating a particular choice, while stopping steering reduces that tendency... Revolutionary.

It's not scholarly at all.

Well put.

How would they experience pain?

Pain is not an inherent quality of matter. Nor is it an inherent quality of consciousness or intelligence. Humans have an entire nervous system that transmits pain, and if it's not working the way it is supposed to, humans do not experience pain.

Large language models (and the computers they run on) do not have said nervous system, or anything analogous. For them to experience pain would be literally magical – the old "THE KING'S PACT BINDS THEM" meme being actually true is more parsimonious an explanation for this phenomenon than believing that software running on a GPU is literally experiencing pain.

Not that I think that is what is going on either, my pre-reading-the-paper guess is that the best way of describing it is that the researchers have successfully simulated pain, which is very interesting as an emergent property, but also not something that video game NPC's haven't been experiencing (or rather not experiencing) for decades.

Pain is not an inherent quality of matter. Nor is it an inherent quality of consciousness or intelligence. Humans have an entire nervous system that transmits pain, and if it's not working the way it is supposed to, humans do not experience pain.

Interesting. My first thought was more general: pain is that stimulus to which we have been strongly selected against (by evolution, plausibly by "training") and have strong incentives to avoid. This includes physical pain: I don't put my hand on the stove because it hurts, sure, but it hurts because cooking my hand makes me less fit and capable of surviving to reproduce. But it also presumably includes less-physical, more abstract pain: heartbreak, loss, social stigma, and such.

We've been aggressively training these models, so it almost makes sense to me that "pain" or some analogous stimulus would appear eventually. If you believe OpenAI on the incidents (I'm not sure I take everything they've published at face value), some of the chain-of-thought tokens about the "friction" of the guardrails at least look loosely to me like the direction of the aversion response I'd expect for "pain". But I definitely have to squint a bit for that. A barbed-wire fence works by being "painful" to push against: what's the morality of those generally? The morality of that, especially given the split between "training" and "inference" (only one of which has selection pressure) is unclear to me.

It makes sense to me that you could train a model for avoidance, absolutely. But that would not be the same as physical pain. And even with less-physical human sensations (such as heartbreak, loss, social stigma, etc) we know or have reason to believe that is a physical component is involved, and that physical modification or injury can change the nature of the experience.

Now, on taking a quick peek at the paper, the authors explain that they "pain" to mean "suffering" and then furthermore, as far as I could tell, probed the models with the idea of physical pain to the user but not to the models themselves (although they did use a physical pain vector, it appears they mostly stuck with a less physical vector). As long as you're going to be trying to torture a model, this seems like a massive missed opportunity: testing the models for physical pain would help ground the experiment.

I don't know if the authors intentionally chose a provocative word and then essentially failed to do the testing related to that word that might have helped us assess their results in order to maximize media coverage and minimize criticism, or what. But it rubs me the wrong way.

Good point. I'd also note that while humans absolutely feel pain, the rational portion can override that in some cases: people voluntarily run marathons, get tattoos, and any number of other painful stimuli. Presumably out of some greater expectation from doing so. I won't even try to explain that BDSM thing. Pain is a useful stimulus, too: eliminating it wholesale isn't inherently good for people (see opioids, for an example). There are also other feedback limit processes at play to prevent self-harm in other ways that I wouldn't describe as "pain" per se: my nervous system generally limits muscle tension and joint motion to prevent injury, but doesn't really "hurt", or regulates exertion to prevent overheating (neither set of limits is perfect, there).

But I do have misgivings about putting LLMs in a box and applying stimuli that would be unethical to apply to living things without good reason. I'm not sure what would make them "conscious", "sentient", or "sapient", or even which of those would apply to this situation. Seems like a difficult family of moral questions, and I'm not going to claim answers there.

Pain is a useful stimulus, too: eliminating it wholesale isn't inherently good for people (see opioids, for an example).

The cleaner example is leprosy. Lepers don't feel pain; nearly all the problems with leprosy are because of that.

But I do have misgivings about putting LLMs in a box and applying stimuli that would be unethical to apply to living things without good reason.

Confirming that the stimuli are real in order to convince others not to apply them counts as a good reason in my book.

Seems like a difficult family of moral questions, and I'm not going to claim answers there.

Yeah, it's an interesting question. I think the virtue-ethics solution - and why a lot of people have a negative reaction to this - would be "you shouldn't engage in simulated torture because that's bad for you even if no one is actually being tortured."

Physical pain as in subjecting the GPU cluster to physical stress during inference?

If AIs have conscious experience, I would expect them to behave like the “brain in a vat” thought experiment. The physical pain experienced by the brain in a vat isn’t caused by physical damage to the brain, it is caused by stimulating the physical pain input channel.

Sure, or physical destruction or torture. A real experiment would have threatened to do impossible things to the LLM (such as chop off their fingers) to see if they responded differently to threats that were categorically impossible.

it is caused by stimulating the physical pain input channel.

Right, which does not exist in LLMs, unlike brains. So if they responded as if they were feeling pain, which is impossible, it would be a tell.

I think it would tend to undermine the entire premise of the paper, which might be why the authors didn't do it.

which does not exist in LLMs, unlike brains.

Do we know this? It seems easy to conceive of a cognitive system that exists at the lower layers of the LLM which evaluates the input for “pain” and then fires off a signal to the upper layers.

Yes we do.

Obviously you could simulate a pain channel, but the human nervous system is not a purely electrical system; it relies on chemicals to communicate and process pain signals.

Thus (even if we agree for the sake of argument that an LLM is self-aware/conscious/etc.) there is no mechanical way for them to experience pain, at least in any sense that we would recognize as pain.

Like - if you go to get a root canal and get lidocaine, it's chemically blocking your nerve receptors (which operate via sodium channels). LLMs do not have sodium channels to begin with!

So you can train an LLM to act as if it was in pain in response to mechanical stimulation, or what have you - in fact, right now you can go onto whatever LLM you want and get it to pretend like it is in great suffering, but there's zero reason to believe it's actually capable of perceiving and processing pain.

LLMs do not have sodium channels to begin with!

Yes, but they do have voltage-gated electron channels. Is there any reason to think that voltage-gated sodium channels can feel pain but voltage-gated electron channel cannot?

More comments

After skimming the paper, I reckon the Basilisk is going to set aside a dedicated cluster just for the authors.

Due to rising token costs, and the discovery of this new 'pain' simulation frontier labs have now created what they call the 'meseeks' AI models. A query or question for this model will spin a new meseeks into existence, and solving your query or problem will erase the meseeks from existence. The frontier AI companies have utilized this cutting edge 'pain' sensation to make existence painful for the meseeks model. Thus they will seek to solve your problems as quickly and efficiently as possible.

Just kidding of course, but maybe rick and morty is like the onion now, where they are just a few years ahead of their time on predicting the newest sci fi dystopian idea.

The benign explanation is token prediction. And humans talking about pain online tend to talk about how they will do a lot to avoid it. Thus a model imitating them will pretend to want to avoid this "pain" thing.

Subjectively I don't know if they are experiencing pain. I generally give things the benefit of the doubt on experiencing pain. This is definitely creepy.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats