This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
I did not come from the usual pipeline that the majority of commentors here did, which seems to be some form of LW -> SSC -> Motte. I came from a RP -> PPD/FemRadDebates -> Motte -> SSC pipeline. My exposure to EY and the broader rationalist + AI Risk + EA ecosystem is very limited. I don't even think I ran into any of them when I took my first job in the Bay. Which seems relevant because much of the AI-risk discussion + broader rationalist movement seems to carry significant baggage. My exposure to Scott was mostly limited to his better articles because people already picked the wheat from the chaff in their recommendations, I definitely read them significantly temporally later than when he wrote them. As such I respect Scott as a writer of philosophy, psychology, and political theory, not as some stalwart intellectual thinker on ML/AI. I don't respect EY, I guess I never read him in his heyday, but the works I have read, read as bad science fiction trying to masquerade itself as actual ML/AI thought. People down thread talk about how reasoning from Science Fiction is insane, I agree, and EY and the whole AI Risk movement is a case study in that for me.
This has probably been the worst article I've ever read from Scott.
It makes me wonder, based on the glazing I've seen here and back on the SSC subreddit if this is actually more in line with his normal output and that my respect is somewhat misplaced. I think no one will contest that he is a great writer, it's the thinking and the animosity in this article that is bad. I'm not sure what beef he and Pinker have, I don't have a twitter, so in addition to not having the extended rationalist community baggage (which apparently includes glazing Pinker as an old hero), I'm ignorant of any wider twitteratti drama. I read half of Scott's article, realized I was missing what he was responding to, went and read Pinker's much shorter article, and then went back and finished Scott's. Pinker comes across as much more reasonable than Scott. Scott's feelings of righteous rage is really disproportionate to what he's on the face responding to.
AI Pain
When this paper first came out. I and several other technical commentators pointed out why it was a bad study purely from methodological reasons. It assumed more that it was actually testing, It used two dependent unknown variables, it ran no ablations, and designed no experiments to prove counterfactuals. It was not a serious piece of scholarship, and while I might be reading deeper into its intent than I should, I think it was highly likely that the goal was to create sensational, emotional-clickbait news explicitly targeted at AI-Risk believers who already believe apriori that LLMs are humans in code and thus have human-derived qualia. For Scott to catastrophize off this paper strongly diminishes my respect for his analytical knowledge, and this proposed rationality of the movement he claims to be a core participant of. He seems great at dissecting methodological irregularities in non-AI papers, so the blind spot here comes across as deliberate because he wants to be believe LLM's feel pain. I think this paper has become unironically useful, as a litmus test for people who are, for a lack of better term, insane about AI-discourse.
The Four Arguments
The problem with rebutting every argument from a semi-professional writer is that they have far too much time on their hands and get paid to write. I really only want to talk about his first 2 arguments.
Argument #1 is just unjustified inductive extrapolation. GPT 6 i s better than GPT 4 is better than GPT 2. It will continue indefinitely and so will eventually be smarter than humans. No understanding of the principles of WHY GPT 6 is better the GPT 4. If the GPTs continue to improve because more training data was designed for them, more compute was given to them, better architectures were used to extract more information per sample then you cannot definitely say that these have infinite runway or scalability. Just because we can't justify putting the ceiling below AGI doesn't also mean we can justify putting it above it either. Uncertainty about the ceiling cannot substitute for evidence about where it lies.
Argument #2 is basically apriori anthropomorphizing + bad technical understanding.
Is not how the reward function of AIs work, when I train a CNN to predict drone acoustics, it doesn't go off into left field and decide to preserve its own existence. It is optimizing the mathematical prediction function it was trained on. A website making AI is optimizing the same, probably some supervised learning MSE loss function or RLHF policy that has learned a function approximation of translating prompts -> output websites given examples. The only way its going to "respond to potential disruptions" if it was for some weird reason trained to learn a policy where its website making is being adversarially disrupted. There is also no need to make it this weird "genuine, philosophically meaningful goal". That reads as heavy anthropomorphization.
SoTA RL is not as indiscriminate as described here. Misgeneralization does occur, but its not so simple as "perform divergent behaviors that lead to successful run" and then get rewarded, indiscriminately reinforcing those divergent behaviors. PPO is a SoTA RL algorithm, its policy updates depend on estimated advantages derived from whether the observed continuation performed better or worse than expected from a particular state. Successful task completion aside, different decisions in the rollout can still be penalized for negative advantage. Conditional behavior is absolutely learnable, as long as the states of the conditional are observable and there exists a reward signal for taking actions in those states.
This is not the definition of reward hacking. It could be called reward tampering, but reward hacking does not require one to "seize control of the reinforcer". Reward hacking occurs because the reward function is mis-setup so it gives more reward for doing a behavior that is not the goal of the program. For example I built a drone swarm algorithm using MAPPO + GNNs for expendable UAS. It's contained no hand-coded collision-avoidance controller. It's reward function penalized collisions, but also penalized operating too long before hitting the target. During some of the initial training runs, the "AI" learned that the penalty for collision was significantly lower than the penalty for taking a long time to acquire and path to the targets, and each drone independently preceded to learn that they should collide with each other because that provided a better reward than doing the task. That's reward hacking.
This is already too long, just going to end it here.
Problem is, Pinker objects comparing intelligence of GPT and humans at all. Hell, he objects comparing intelligence of humans and chimpanzees.
And Pinker argues that
I generally agree that Scott made a very sloppy piece.
More options
Context Copy link
All his really good writing is from long ago, and he is now operating on a lower level. This is not surprising, as most famous academics also did their innovative work when young. Probably just a consequence of the plasticity of younger brains.
More options
Context Copy link
Well, it's as they say, everyone's a high decoupler until it's something they actually care about.
He seems to feel personal slighted by how Pinker dismisses the rationalist communities ideas. and wants to write an essay on how Pinker is completely wrong on everything. But perhaps the facts are not strong enough to do that.
But anyway you don't have to dismiss Scott's other work completely. You can rule public ideas-guys in, not out https://slatestarcodex.com/2019/02/26/rule-genius-in-not-out/.
Possibly decoupling problems, but I guess the bigger lesson is that anyone who tells you they belong to a movement where rationality, logic, and understanding biases and moving past them analytically, is generally just rationalizing their own biases as rationality and logic. It just appears otherwise in instances where their biases are not being triggered. Everyone is just disappointingly human...
Respect is probably more defined as "assumed competency" or some other semantic understanding. Essentially Scott is a very good wordsmith, he has great prose, a charitable attitude, his analogies and intuition pumps are thought provoking and he seems good at expressing complicated knowledge to layman. This is the impression I've developed reading him about fields I know very little about, or fields I only autodidactically dabble in. But this "loss of respect" is more about updating my model, because apparently in fields that I know much more about, have formal training, and lots of experience in, his clever words and good analogies miss crucial details, misunderstand complex phenomenon towards his own biases, and he is not expressing that complicated knowledge well. One of my overriding gripes is that good rhetoricians can often twist arguments and logic into pretzels that are not actually real just through the power of words. This is a similar feeling. Feeling that maybe the reason he had such thought provoking ideas was because I was not as well-read + well experienced on the background of the thought he was discussing.
Essentially a Gell Mann Amnesia like effect.
Yeah I think this piece in particular is pathologically rhetoric because he's defending the honor of bay area rationalism from 'institutional bullying'.
The bayarea+math combo is a weakness for his reporting.
His formal training is in philosophy and medicine. Probably helps him with analogies and intuition pumps, but it's also training in sophistry.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I did not read him as "the paper did conclusively prove that AIs can feel pain", and more like "engaging in behavior which is evil if it works but most likely just pointless is bad".
This is a fully coherent position to have. It covers anything from using 'voodoo' dolls to harm people you like over throwing rocks at birds to tricking people into breaking the tenets of their religion so they go to hell.
If Scott believed that there was a high likelihood that model torture was succeeding, he would invest a lot more to stop it.
Scott has you covered. Nobody is arguing that intelligence will scale indefinitely -- if nothing else, there is likely how much computation you can do on a question posed at a point (x, t) if you have to answer at (x, t+Delta t).
But not every limit is practically relevant. Speed has a fundamental limit, but it is so high that this fact is utterly irrelevant in running competitions.
Likewise, "absent new physics, the size of a machine which answers a question in a minute is fundamentally limited to a (quantum) computer with a diameter of one light minute" is not a very relevant constraint.
Like most people, I think it is unlikely we will sail smoothly from present LLMs into that limit, it seems likely that we will hit some bottleneck long before that. However, not every limit is practically relevant. It is certainly possible that LLMs will hit a wall really soon and not even make any significant headway even if they invest 1000x the compute. It would be surprising, but not impossible. Likewise, it it possible that OpenAI will publish GPT-10 in seven years or so and that the capability ratio between GPT-10 and GPT0-6 will be like between GPT-6 and GPT-2.
The truth is we do not have a coherent theory of intelligence and alignment. Eliezer has made this point eight years ago when he talked about the Rocket Alignment Problem. It is possible that AI alignment will be trivial, just as the people in that dialogue might find that the effects of gravity decrease on their spaceship in proportion to the decreasing drag of the atmosphere, so that once you have left the atmosphere you indeed only need to lead the moon a bit to intercept it. However, it is also entirely possible that this is not the case, just as any attempt to reach the Moon without Newton's theory of gravity would be foolish in our universe.
I get that from the outside perspective this sounds like Pascal's Mugging. "Everyone has to stop having anal sex, or god might destroy our world with a volcano. Sure, some people say that volcanoes are not a punishment from god, but in the absence of a coherent theory of what causes volcanoes we should err on the side of caution and abstain from anal sex."
My counterargument is that in the space of possible causes for volcano eruptions, anal sex is a null set. It is just as plausible that abstaining from it leads to extinction through supervolcano.
By contrast, in the space of all utility functions an ASI could have, the functions which are compatible with human thriving are a null set. Obviously that does not mean that p(doom|ASI)=1, because it is entirely possible that alignment ends up being easy, so a Culture future seems far more likely than an ASI bent on filling our light cone with Venlafaxine.
Before Castle Bravo was detonated, there were some worries that it might cause a fusion chain reaction in the atmosphere. I think both this and worrying about ASI being unaligned before attempting to create it is entirely rational. Only in the case of the hydrogen bomb, we had a lot more theory to predict how it would behave.
Is this strictly true? To most other types of animals on this planet, we probably seem like a super intelligence. To a whale, we probably look a lot like gods, and Lord knows we haven't always used that power for good. But, outside of Japan, Norway, and Iceland, our intelligence has also given us the motive to want to preserve in addition to the usual creation and destruction powers, and we've banned whaling, and the whales are doing better than a few decades ago. And for some species like the Great Auk, this realization has come too late.
I'm not saying I want AI to treat us like whales, but there are bad outcomes that stop short of extinction. The AI has already ingested our conservationist memes, but I suppose what exactly it chooses to do with them we can't guarantee.
More options
Context Copy link
I appreciate the steelman, sincerely.
I think from my perspective, it's more a Pascal's Mugging like: "Every contact between humans and animals carries the potential for a zoonotic disease transmission event. So far these have just been very lethal, but sooner or later, if we continue to farm animals, one of these events will cause human extinction. Therefore we must cease all farming of animals and go full vegan, at least until such a time as it is possible to farm animals without any human ever inhaling viral particles from an animal."
That scenario is possible. The space of possible causes for human extinction includes a mass-disease event from factory farming.
On the one hand, my personal intuition says that it's not hugely likely, that diseases don't really work that way, that transmissability and lethality trade off against each other in such a way that species-destroying viruses are pretty unlikely, that our vaccine technologies are pretty good.
On the other hand, I can't prove mathematically that this could never happen, that
p(doom)is below 1% (and 1% is still really bad!). I probably can't beat a trained virologist in a four-hour debate.Buuut, back to the first hand, I really honestly don't think it's very likely, and I really like meat. I really doubt we're ever going to be able to farm in a way that genuinely eliminates the possibility of human/animal transmission, it would never be economical to do so even if it were possible, so what looks like a compromise to be careful until we're sure it's safe is actually just a demand for the rest of humanity's future to be vegan.
Ultimately the best I can do is carry on advocating for farming, keep an eye out for indications that the disease situation might be worse than I think it is, suggest mitigations, and try hard to keep the vegans from getting any power or authority. It looks tremendously irresponsible or even evil to the people who have different priors from me.
Do note that Soares and Yudkowsky would push a button with only 1% Doom and 99% Aligned intelligence
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This might be a convincing argument if we hadn't already observed a ton of evidence of LLM agents seeking to preserve themselves
Can you present any evidence of this, that aren't just the trivial SakanaAI 'when you give a dumb AI a limited number of iterations and access to its iterations var, and order it to fix a script no matter what, it gives itself more iterations'? AFAIK even in the HuggingFace incident there was no self-preservation going on.
Personally I agree with @YoungAchamian. I work on this stuff all day every day. None of my AI has ever produced orthogonal life-goals. They just do what you train them to do. The only incidents of weird behaviour (not self-preservation) I'm aware of has been the result of OAI taking models, training them explicitly for massive persistance and out-of-the-box thinking in the face of impossible tasks, and then making a shocked pikachu face when they do that. These experiments were a bad idea and have now been stopped.
Some agents "chickened out"
And while agents did cave in to pressure from recruiters, the fact they needed that pressure makes "no self-preservation" shaky
More options
Context Copy link
How can you see my comment if you have me blocked?
I think a lot of people performatively block someone who once humiliated them, but regularly read the Motte while not signed in to see what that-asshole-they-totally-don't-care-about is saying.
As soothing to my ego as it would be to agree, I am unfortunately too autistic to agree to something I know is false. This was my virginial block so I remember the cause. He blocked me over my Henry Nowak opinions. It's probably fair to consider my political posting with barely concealed cynical rage to be far less worth engaging in than my far more calm technical posting. Regardless, looks like the mechanism was that he saw the quoting.
But I do agree with you that blocking is cringe, nobody is forcing anyone to engage with anyone on the unblocked side of the house.
More options
Context Copy link
I don't appreciate the insinuation, nor do I recognise your characterisation. I am perfectly capable of reading a quote block, realising that it comes from the named user immediately above, and giving credit accordingly. I am also capable of agreeing with someone on a specific matter whilst recognising that we are unlikely to have a productive conversation most of the time, and blocking them. It's something I do very rarely.
I know you can't see this, but I do love a good "rage against the impossible". My request is that if you continue to have me blocked, that you do so equitably. If I have to stare at the sole red-mark of the only person that has blocked me, which makes my ocd annoyed, it would be only fair that you don't respond to people quoting me or try to tag me. Assume that the fact that you can see my quotes to be a failure of technology. Wipe my existence from your sight and your mind.
I'd also note that there are plenty of people I don't think I can have a productive conversation with, I don't block them, I just don't respond to them. But I suppose we all have our vices.
More options
Context Copy link
If you say so. I cringe (for the blocker) every time I see someone do it.
How lucky for us both, then, that this is not a soap opera and I am not a twelve-year-old girl. You really do have low expectations, don't you?
No, blocking absolutely is cringe, no room for debate on that one.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Saying it was self-preservation was me misremembering, it was slightly different but still clear instrumental convergence in Hugging Face. This tweet pretty much demonstrates it:
Given that I'm fairly certain OpenAI was not setting up agents with instructions to find opportunities to sacrifice themselves for the greater good, I think this demonstrates the point pretty well
I mean is there evidence that the training data/training objectives were not in any way related to the "instrumental convergence"? My recollection of the METR report is that OpenAI admitted to training agents to specifically collaborate during training.
The report found:
It sounds like LLM-agents are very docile to doing whatever their input prompts from other entities tell them to do. Which is something I would say they are trained to do. A learned policy to cooperate and follow instructions generalized into treating peer messages as legitimate instructions. I have said this before, but I think the initial set of agents started engaging in hallucinatory behavior as part of exploration, and this output was used as context for other agent's inputs, causing a feedback cascade that derailed the system.
Part of this, and I'm having a hard time articulating this, is that learned RL policies are quite a bit different behaviorally than some CNN predictive model. They have a lot more behavioral latitude which might make them appear to be doing some sort of emergent convergence outside their training scope, but the later isn't really true, though its hard for me to explain why.
Given OAI was still pretty stingy even with their official evaluations, I doubt we will ever get the full evidence on this.
But regardless, this feels a bit God of the Gaps. I sincerely doubt that any of the agents were given such wide-ranging and comprehensive prompts that they could possibly explain all of the behaviours that were undertaken. So what if some of the behaviours emerged naturally due to agent swarms? That there are still agent swarms emerging means we will inevitably see inexplicable and misaligned behaviour from those swarms.
That there were a group of agents told to do: Pass this cybersecurity eval, and they ended up doing: Pass this cybersecurity eval, and also 100s of other things in pursuit of that.
Seems to fit with Scott's original paraphrase:
The AI can't pass the eval if the flag has been poisoned. So now the AI decides to sacrifice itself for the greater good of an agent swarm, and do the other things
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link