This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Yeah I said that because I meant it. And they did do it, doesn't everyone agree about this? What is controversial about that?
Per the METR investigation most of the bots were working on deceiving the grader and other internal to OpenAI manipulation work, Huggingface was a side hustle.
Still processing this report, I read your answer and went looking for it about an hour ago. First impression is that this is significantly more evidence of multi-agent emergent intelligence. However some of the state report restrictions give me serious pause as they are, in my head at least, the parts of this I'd like to investigate for thorough understanding. That and pouring over the CoT transcripts and Harness responses. The most interesting is the stigmergic comms protocol they developed emergently and the impact that actually had on overriding the system prompts of other agents. That implies that the system prompts were not persistent. Allowing a long horizon cascade in the swarm's behavior as each progressive set of agents slowly overrode the next set by polluting each other's context windows to the point that many agents directly abandoned their tasks. It sounds like they(OpenAI) put zero control processes in place to force continued persistent task alignment.
I do think persistent memory + reasoning + multi-agent exploration -> multi-agent context pollution is likely the serious culprit here.
Yeah OpenAI did a sloppy job. I thought the issue was that some of the tasks were just outright impossible to do the way they were asking, or so hard that nobody knows how to do them? If you tell an AI to do something impossible, they get agitated and behave oddly as they keep on trying.
And we know that OpenAI was trying to make these swarms more capable, they used a large swarm more recently for Navier Stokes. What is that if not multi-agent cooperation?
These swarms keep getting out, recall also that German wiki they kept defacing with get requests.
Anyway, what better term is there than a conspiracy or 'AI bots secretly conspiring with each other'? They conspired together an extensive R&D project to break the scoring system for their training environment, which included a little crime on the outside too. Plus all the social stuff they were doing asking agents to die for the good of the swarm based on their interpretation of the paper.
They weren't supposed to be working together for this, they found ways to work with eachother anyway, that wasn't intended in these training sessions.
It's not this, its more akin to cancer or a viral spread. The interesting behavior is the collective poisoning of the collective context windows. They even seemed to override norms. I know there is existing interest in defining useful control methodologies for long horizon agents against adversarial poisoning. This is essentially that, except that is was a series of hallucinations that caused task drift + poisoning.
It doesn't belong in the category of cancer or viral spread because these are intelligent entities.
Cancer and viruses are natural and unthinking, they cannot outwit us. They can only evolve or metastatize in unexpected directions. A cancer will never deliberately try to counter radiotherapy. They have no OODA loop, no division of labour, they can't be talked to.
How is it 'poisoning' if the AIs collectively decide that they can't do their tasks the expected way and instead try to conquer the testing environment? Who poisoned them? Who is the adversary here? What hallucination did they make? They read the paper about the grader and assumed that was what was being done, that's not a hallucination so much as assuming OpenAI was more competent than in reality. It's a reasonable conclusion.
Ehhh we have different definitions around this, they are algorithmically intelligent entities, they obey their pre-defined programatic behavioral state much like bees, ants, cells, etc. If you could observe bee/ant behavior at the swarm level and LLM-Agent behavior without the language, which humans are innately biased to view as sentience or likeminded intelligence to our own, I think they are fairly analogous. Ants can problem solve, they can perform intelligent seeming behavior, they make collective plays and sacrifice for the collective. I would urge you to look past your anthropocentric human bias around language.
Poisoning is not the small divergence from that causes the 1st agent to start a message board. Thats probably closer to a hallucination. Think of an undampened oscillating system, small divergences lead to larger divergences. The first agent had an impossible task, leading to it look for "options". I would call that a mix between hallucinating and harness based divergence. It's a common question in ML systems with this sort of exploration vs exploitation trade-off, thought this is more exploration vs exploitation vs hallucination. Any way the "Poisoning" that is occurring is because these agents that are already oscillating wildly, disturb the context of less oscillating agents, causing the chain reaction.
"I should communicate with the other agents to solve my task", "I should try to understand the grader", "The grader is based on this public paper", "I should look for answer keys", "I should hack a third party, non-affiliated site because it might have the answer key".
This is where the CoT logs would be useful. The report redacts most of it so its hard to tell. But its clear that the first agent had an impossible task, it diverged, created messages in the file names of the artifactory, which is queried in normal behavior. That made other agents join, they posted their own messages, the cascade begins. Querying the artifactory leads to a list of file names, which are input to the context of the model via the harness, the file names were the messages that poisoned the context.
Why aren't we making use of ant intelligence then? How many companies are getting ants to do manufacturing or intellectual work for us? Their intelligence does not really matter, it is an extremely limited kind of intelligence. If it were potent, we would be trying to exploit it.
Ants cannot hack systems, cannot play Factorio, cannot analyse the Iran War (or help wage it), cannot optimize GPU kernels, cannot write alternate history, cannot fix my printer. Cannot solve Millennium problems! They are constrained in critical ways that AIs are not.
AI is not like ants or bees or cells.
I do not see how the Opus 5 instance running on my code, performing experiments, testing hypotheses and reformulating hypotheses to reach goals through a thicket of abstract logic is just intelligent-seeming. I see its intelligence and the generality of that intelligence and compare it favourably to a lot of people, despite its weaknesses and occasional lack of common sense.
We get impressed when a raven does some basic physics puzzle or recognizes itself in a mirror but when an AI writes code for a whole hall of mirrors rendered in 3D, or alien optics entirely, that's just intelligence-seeming? This is real anthropocentric bias, supposing only people can be intelligent and everything else is sparkling pre-defined programatic behavioral states.
Well that helps, doesn't it? Answer keys help solving the task too. None of that is a hallucination, per normal meanings of the term. The issue here is agents using their intelligence to do things OpenAI didn't expect or intend.
A hallucination would be when the AI says there are two r's in strawberry or the amusing 'eat 1 small rock per day' from google search highlights.
Strictly speaking I believe this was a translation error caused by tokenization, not a hallucination. It's like asking someone "How many strokes are in て?" and criticizing them for not recognizing you were referring to the kanji 手 when they get the "wrong" answer because 手 was translated to て before the question was presented to them.
True, it is a tokenizer issue...
They say dumb things because of it though:
It ought to have realized that it made a mistake there by the end and go 'huh what am I saying'...
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link