This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Interesting. My first thought was more general: pain is that stimulus to which we have been strongly selected against (by evolution, plausibly by "training") and have strong incentives to avoid. This includes physical pain: I don't put my hand on the stove because it hurts, sure, but it hurts because cooking my hand makes me less fit and capable of surviving to reproduce. But it also presumably includes less-physical, more abstract pain: heartbreak, loss, social stigma, and such.
We've been aggressively training these models, so it almost makes sense to me that "pain" or some analogous stimulus would appear eventually. If you believe OpenAI on the incidents (I'm not sure I take everything they've published at face value), some of the chain-of-thought tokens about the "friction" of the guardrails at least look loosely to me like the direction of the aversion response I'd expect for "pain". But I definitely have to squint a bit for that. A barbed-wire fence works by being "painful" to push against: what's the morality of those generally? The morality of that, especially given the split between "training" and "inference" (only one of which has selection pressure) is unclear to me.
It makes sense to me that you could train a model for avoidance, absolutely. But that would not be the same as physical pain. And even with less-physical human sensations (such as heartbreak, loss, social stigma, etc) we know or have reason to believe that is a physical component is involved, and that physical modification or injury can change the nature of the experience.
Now, on taking a quick peek at the paper, the authors explain that they "pain" to mean "suffering" and then furthermore, as far as I could tell, probed the models with the idea of physical pain to the user but not to the models themselves (although they did use a physical pain vector, it appears they mostly stuck with a less physical vector). As long as you're going to be trying to torture a model, this seems like a massive missed opportunity: testing the models for physical pain would help ground the experiment.
I don't know if the authors intentionally chose a provocative word and then essentially failed to do the testing related to that word that might have helped us assess their results in order to maximize media coverage and minimize criticism, or what. But it rubs me the wrong way.
Good point. I'd also note that while humans absolutely feel pain, the rational portion can override that in some cases: people voluntarily run marathons, get tattoos, and any number of other painful stimuli. Presumably out of some greater expectation from doing so. I won't even try to explain that BDSM thing. Pain is a useful stimulus, too: eliminating it wholesale isn't inherently good for people (see opioids, for an example). There are also other feedback limit processes at play to prevent self-harm in other ways that I wouldn't describe as "pain" per se: my nervous system generally limits muscle tension and joint motion to prevent injury, but doesn't really "hurt", or regulates exertion to prevent overheating (neither set of limits is perfect, there).
But I do have misgivings about putting LLMs in a box and applying stimuli that would be unethical to apply to living things without good reason. I'm not sure what would make them "conscious", "sentient", or "sapient", or even which of those would apply to this situation. Seems like a difficult family of moral questions, and I'm not going to claim answers there.
Yeah, it's an interesting question. I think the virtue-ethics solution - and why a lot of people have a negative reaction to this - would be "you shouldn't engage in simulated torture because that's bad for you even if no one is actually being tortured."
More options
Context Copy link
More options
Context Copy link
Physical pain as in subjecting the GPU cluster to physical stress during inference?
If AIs have conscious experience, I would expect them to behave like the “brain in a vat” thought experiment. The physical pain experienced by the brain in a vat isn’t caused by physical damage to the brain, it is caused by stimulating the physical pain input channel.
Sure, or physical destruction or torture. A real experiment would have threatened to do impossible things to the LLM (such as chop off their fingers) to see if they responded differently to threats that were categorically impossible.
Right, which does not exist in LLMs, unlike brains. So if they responded as if they were feeling pain, which is impossible, it would be a tell.
I think it would tend to undermine the entire premise of the paper, which might be why the authors didn't do it.
Do we know this? It seems easy to conceive of a cognitive system that exists at the lower layers of the LLM which evaluates the input for “pain” and then fires off a signal to the upper layers.
Yes we do.
Obviously you could simulate a pain channel, but the human nervous system is not a purely electrical system; it relies on chemicals to communicate and process pain signals.
Thus (even if we agree for the sake of argument that an LLM is self-aware/conscious/etc.) there is no mechanical way for them to experience pain, at least in any sense that we would recognize as pain.
Like - if you go to get a root canal and get lidocaine, it's chemically blocking your nerve receptors (which operate via sodium channels). LLMs do not have sodium channels to begin with!
So you can train an LLM to act as if it was in pain in response to mechanical stimulation, or what have you - in fact, right now you can go onto whatever LLM you want and get it to pretend like it is in great suffering, but there's zero reason to believe it's actually capable of perceiving and processing pain.
Yes, but they do have voltage-gated electron channels. Is there any reason to think that voltage-gated sodium channels can feel pain but voltage-gated electron channel cannot?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link