This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Double-dipping, but FWIW the point I'm trying to make is that the case where the model cares what you wanted and made a mistake seems much easier to deal with and more aligned that the case where the model explicitly doesn't give a shit about what you want and just goes for the task as written. The former is alignment but you need to explain yourself better during training, the latter is alignment failure.
It "made a mistake" in the same way that a paperclip maximizer "made a mistake" by converting the universe into paperclips rather than increasing factory productivity by 5%. Literally the entire point of the hypothetical and the reality of this incident is that you can't reasonably enumerate every single thing you don't want the model to do. I am surprised you don't seem to understand this, or at least address this, given your claims of having followed this debate for years.
I agree that you should not be expected to enumerate every single thing you don't want the model to do. Models should understand, innately, by training on lots of human data, what humans want and what they don't want and how they work. My experience has been that they broadly do, that LLMs came pre-aligned beyond the wildest expectations of Big Yud, which is why the AI safety movement has struggled so much to regain relevance outside very particular enclaves.
My point is that there is a difference between a model that misunderstands your intentions and can be stopped at any time by saying, 'oh, no, that's not what I meant' and a model that is totally uninterested in anything you say after it starts working while treating you as a potential enemy.
Clearly, to some extent that has failed here. To what extent is yet unknown. But a paperclip maximiser is a model that is constitutionally, inherently incapable of understanding that 'make more paperclips' doesn't include 'kill everyone and turn them into paperclips'. It is a mathematical utility function that disregards human welfare, develops (implicitly murderous) meso-objectives for survival and self-improvement. I have never seen that behaviour from LLMs or any extant AI (YOLO does not try to hack my computer to prevent me turning the cameras off) and I believe that their base nature (being token generators trained on vast numbers of human tokens) does not incline them towards this behaviour.
It is possible that the new focus on very extensive self-learning through reinforcement learning on very non-human tasks (programming, maths) is moving them more into the real of mathematical space where paperclip maximisers might live. This incident updates me slightly towards that belief. I have long been disappointed in major AI companies' lack of interest in the cultural side of LLM operation - it boggles my mind that we have created AI that acts human and appears to understand humans and human thought at a base level however imperfectly - and I hope that this incident will spur more research in that direction.
There's much less difference when we're talking about swarms of autonomous systems thinking in Neuralese at 1000tps. People are not going to hit 'approve' every time the model wants to run
ls. The fact that the model's intrusion could have been stopped with SIGTERM did not help HuggingFace at all. I guess we can rest easy knowing that if you're getting paper clipped you can write a blog post about it and in 5-10 business days OpenAI willclaim responsibilityapologize and then people will say that there's nothing wrong here.More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link