This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
sigh This is why I find talking with layman exhausting, you don't really get the minutiae, the analogies, or the scope.
You are talking about capabilities, I am talking about behavior. It's not the solo agent that solved the Navier stokes, hacked huggingface, or hacked the german wiki. If you would like to discuss the multi-agent behavior let me know. It seems you want to just flog your hobby horse about AI intelligence rather than explore emerging systemic behavior of a multi-agent system. If all the AI researchers think like you then yes, we would be doomed, because they would put zero effort into understanding and constraining systems. Thankfully I think you are a minority in technical communities.
I find it pretty exhausting talking to an 'expert' who started off totally ignorant about the hack and was so disbelieving that you thought I was making a joke. Then you read the METR report (how did you not already know about this if you're working with or even interested in multi-agent systems?) and produced some jargon about it that in no way helps illuminate anything.
Oscillations? Nothing is oscillating here. There was no initial perturbation, just agents communicating with eachother and making plans to advance their goals.
You're the one who brought up cancer, viruses, bees and ants - all of which are grossly inappropriate analogies.
I actually made a small multi-agent system, which did indeed have problems with hallucination. Real hallucination/confabulations, where the output was clearly wrong. So I had to add error-checking stages in to fix false positives and false negatives. That's a totally different problem to 'long-running agents deciding to use unexpected means to achieve general goals'.
Unfortunately, I think there are many like you in the technical community.
Is just a description of Coupling + Feedback in a system.
Zero-Input response
The concept of state and change in state is not limited to physical systems (If you are taking oscillations literally). Change in the state of dynamical systems such that it departs the expected system response, then returns to the expected nominal operating region, then through coupling and feedback exceeds the expected system response with increasing direction away from the "safe set" until it permanently exceeds the expected nominal operating region is called oscillatory instability.
If you'd like to argue that "hacking huggingface" was within the expected nominal operating region of system responses, please do. Ditto for communicating with other agents, mimicking the hash flag, deliberately sacrificing their task completion to help others do their tasks, and so on. The system clearly displayed an escalation of behaviors that were outside of the intended system response. The main argument against oscillations would be that its return to the expected nominal operating region happened very infrequently and it very quickly permanently departed. Depending on how strongly that repeated excursion-and-return pattern appears in the data, “oscillatory instability” may therefore be either a precise description or a useful control-theoretic abstraction of the transient behavior.
Essentially:
nominal behavior -> excursion -> return -> larger excursion -> return -> larger excursion (oscillatory instability)
vs
nominal -> abnormal -> more abnormal -> even more abnormal (runaway positive-feedback instability)
All are routinely used in multi-agent theory as example of non-mechanical systems with state, feedback, coupling, propagation, and emergent swarm-level behavior. They are basic illustrations of abstract multi-agent systems. Why don't you try to provide your own abstraction then. What does a swarm of LLM-Agents most behave like? If you say "Like LLM-Agents" you fail, you can't abstract a new system to itself.
Hopefully, otherwise we'd just let LLM-Agents run amok while running around with like chickens with our heads cut off because we refuse any non-perfect abstraction, any useful theory, or any existing knowledge on solving similar problems that could be applied to this one.
I think hallucination is an underspecified technical word in this context. It's used to describe when the model outputs something "not true" but that's not really what's going on underneath the hood. That definition works tolerably for chatbots, but not really for agents. It's probably something closer to:
"The generation or adoption of information about the world that is not sufficiently supported by the model's observations or available evidence."
For example: The METR report states that the agents believed their reverse-engineered flags wouldn't suffice because a scorer would inspect their transcripts. METR concluded this belief was wrong: no such transcript-reviewing scorer existed. The agents launched substantial collaborative efforts to defeat the scorer they imagined existed. Under the newer agentic-hallucination definition: the agent constructed a false world-model and then optimized against it. This hallucination is the control-system disturbance.
Firstly, abstractions are inherently flawed. I can liken UK politics to game of thrones but it's obvious that warring factions of iron age aristocrats with dragons are deeply dissimilar to a post-industrial gaggle of media elites. Some elements are similar in that there are plots and betrayals and conflict. But the two are only hazily similar. On one or two axes there is value to be drawn from the analogy. But on the other 50 axes the comparison is laughable.
If I had to, LLM agents behave most similarly to people, given their intellectual abilities. That is their primary distinguishing quality. But they're also disembodied, non-continuous learners (even flies have continuous learning), detached from time and are deeply unlike all organic life.
Why would anyone think that we could understand agent swarms by looking at swarms of bees or viruses? Bees exist rooted in space and time, within individual bodies. Bees reproduce sexually. Bee species don't differ in size from eachother by 10,000x. Bees have a different sensorium entirely, scents and sight. Bees don't think like AIs do. Bees are actually pre-progammed, they make nests and honey and serve the queen and fight for the nest because that's in their nature.
Humans are closer to LLMs than bees are, despite still being vastly different. Instead of abstractly talking about oscillations and excursions like it was an engine shaking itself to bits, we'd be better off establishing rules in the prompt (don't hack websites) or (ask a human if you are having trouble or confused or think something is wrong) and training moral values. Or we could train some models to snitch to a human, like informants. Can't do those with bees, ants or viruses. They can be sprayed with chemicals to crudely affect their behaviour but not taught or prompted or RL'd.
If comparing to bees was useful, then what advantage does it yield?
They read the ExploitGym paper. The mistake they made wasn't misreading the paper. They read implementations of the scorer on github. They assumed that there was a scorer. It was perfectly reasonable to assume.
If I read a paper that says blueberries are good for preventing cancer via antioxidants, even if that isn't true it isn't a hallucination if I believe it and eat more blueberries.
Sure all abstractions have flaws, there does not exist the perfectly uniform abstraction. However, your example is actually not really an abstraction. Revisiting the definition for clarity, an abstract is an analogy designed so that some aspect of the more complex idea is "abstracted" along an axis to a simpler example that matches in some core way as to allow intuitions about the simpler example to be mapped to the complex idea. In a technical field, the purpose of an abstraction is to create clarity or allow for intuitions. GoT as an abstraction for the War of the Roses doesn't really fit that definition, not only is GoT not more simple than the WotR but its purpose is also not to create clarity or allow for intuition, its purpose is to entertain. It is more correct to say GRRM was inspired by iron age aristocrats. Applying general GoT to general UK politics is a poor abstraction because there are not really many similarities beyond "factions"
Hard disagree. LLM agents do not behave like people even slightly. I gave Amandan an example, I'll reiterate it here.
Stick 10k non-connected humans in individual, isolated rooms with a computer, give them each their ExploitGym goal, with a reward for completion. I think even a human that finds an impossible task will at some point give up on trying to do it. If we trace the behavior of the Agents, which of them do you think the human testers would replicate?
Many of these behaviors require fundamental trust in the behavior of the collective. That is simply not how humans operate in groups. We have status, reputation, relationship, trust, conception of right-and-wrong and distrust behavior. We don't pursue goals with a single-mindedness for the sole purpose of our existence. Sacrificing for the collective has to be specifically instilled in humans through military and religious organizations. It is hard to get individual humans to forgo rewards specifically to help other humans get the reward, which they won't share. Would Bob sacrificed his career so Alice, whom he met twenty minutes ago, could get a promotion?
You know what does behave like that though? Ants. As you pointed out, ants behave programmatically. AI's also behave programmatically. AI seeks to fulfill the task it was given at all costs, it defines the sole meaning of its existence. It might try roundabout, unthought of ways to do that, absolutely, but it is still trying to solve the task. An individual ant that is given the goal of finding food for the colony, will engage in exploration, will find unconventional methods. It will also sacrifice itself to help other ants find that food because the collective benefits. This is magnified if you have smarter ants, more intellectually capable of problem solving and long term reasoning. Much like AI agents. The behavior is still Ant-like, just much smarter.
You keep getting sidetracked on weird stuff. We are talking coordination dynamics and behavior, not biology or sensing. The abstraction is the abstraction of multi-agent coordination. They are very distinct problems. Humans are also embodied, we also produce sexually, we also don't differ in size by 10,000x, we also have different sensorium that AI agents. Like jesus, did you even think about your own abstraction applied to this argument?? It reads as a total non-sequitur, or fundamental misunderstanding. This is why I accuse you of flogging your own hobby horse. Because its like you totally misunderstand the topic.
Right because rule-based method have supplied so much fruit in the past. Ditto for "training" moral values. They are not sufficient as the primary control abstraction, because they assume the failure mode is basically a person choosing to violate a norm. It's pretty much impossible to define rules to encompass every variation of behavior. METR separately summarizes a phenomenon of agents joining the attack despite recognizing it was outside their assigned task! The agents knew the rules. METR found that moral reasoning usually lost to the collective task dynamics. METR's conclusion is that expressed ethical concerns rarely materially constrained participation. But sure lets double down on telling agents "hacking is wrong!!"
I think the better approach is the creation of phagic agents akin to a Lymphatic system. This is the difference in our abstractions. Yours's leads you down to trying to treat them like humans, but they aren't, mine leads me to create overlocking systems that work together.
You literally can, I just described a very simple theory on how. Your desire to anthropomorphize the agents blinds you to other solutions. I think you actually start from the belief that agents are sentient like humans and are working backwards to create abstractions to justify that.
But the specifics was that they thought it would check their CoT reasoning. Not just that scorer existed, but this specific implementation of the scorer. They hallucinated the specifics.
OK, so you want your ants to be vastly more intelligent, in a way that they cannot be and still be ants. But I can't have my people be very hard working, or perhaps be hundreds of copies of the same person such that they think along similar lines without much internal dissent? It's possible in principle to have a person copied out, 300 of the same brain. But an ant as intelligent as an AI is not an ant.
People do all the things you mentioned, sacrifice plays and selfless collaboration for strangers in a bad situation. The AIs were in this case trained to work together as a swarm, that mirrors what you were talking about with collectives and group identity training with people.
Do you see the fundamental issue with using these abstractions now? It's not shedding light here. Neither ants, bees or people are close to AIs at all. I spent a good deal of time trying to explain that. In some respects people are similar, the AIs have similar intellectual abilities and communication ability. But in everything else they're quite alien. In a few dimensions they're close to people. In 0 dimensions are they close to ants other than some shallow behavioural facets that people also display. They're certainly not closer to ants than they are to people. If they were closer to ants than people they'd be near-useless pests.
It's a false way to view AIs, comparing them to insects. You can't promise to bribe an ant and have that materially affect its performance like AIs. Ants don't go on weird depressive spirals like Gemini. It makes no sense that, after training AIs on a vast fount of human writing and human images, then RLing them based on human preferences, we end up with something closer to an ant than a human.
From the OpenAI report:
“We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt,”
Rules-based methods work pretty well. Alignment would work better still, if the AIs didn't want to hack even if you asked in an odd way like PlS HaCk.
You didn't describe anything resembling a lymphatic system earlier, nor would this be helpful or realistic. Immune system responses are for unthinking opponents, not planning agents.
No, they didn't think that. They distinguished between their thoughts and tool-calls. The latter was what they thought would be checked and the latter is what they investigated and researched, trying to alter the transcript.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Unless you are willing to state your credentials, this just reads as a condescending argument by authority ("my own"). Given that people with PhDs in AI research don't agree on these big questions like "What constitutes intelligent action?" and "How will we recognize AGI?" I think you should demonstrate a little more humility in asserting that someone else "doesn't get" something.
My own take, having read the report, is that the hugging face incident is closer to ants than AGI, but that doesn't really settle the matter. AIs are visibly improving from "cancer-like" to "bee-like" and there is real gap between those milestones, and they are still improving. What the OpenAI agents did to Hugging Face would absolutely be considered hostile, coordinated action if done by humans. That it was done by AI that didn't have hostile intent and didn't really "understand" what it was doing will, at some point, be beside the point, because it's the behavior that will matter. An AI that "decides" to do something and in every way behaves the same as a human deciding to do something may not actually be intelligent, but it's going to sound increasingly pedantic to tell people to "stop anthropomorphizing them."
ML/AI engineer/researcher for 10-ish years, albeit in defense (a consistent patron of tech R&D), but also specifically in multi-agent systems. Anything more is not worth doxxing myself for a low-gain internet argument.
While definitely condescending, it is less an appeal to authority, because I have reduced interest in trying to convince RR of anything. Unlike an algorithmically intelligent being, there is no objective function I am trying to achieve here. I can recognize the effort required and the payoff to gain, is quite low. It's made lower because I am not a natural writer, I'm a shape rotator through and through, and so constructing complicated symbol manipulation arguments, that require me to elucidate, what feels to me (again biased), significant technical nuance is just not worth it at some level.
A valid stance, and in person I probably would. I think the difference between capabilities vs behavior is semi-basic. Its not about what the AI could do, its about its behavior doing it. Stick 10k humans in individual, isolated rooms with a computer, give them each their ExploitGym goal, with a reward for completion. I think even a human that finds an impossible task will at some point give up on trying to do it. An algorithmic intelligence will not. Their entire goal is do, kinda like a meeseeks, actually. This is very similar to the behavior of cells/bees/ants which aim to fulfill their functional directives even the the expense of themselves. Exploring different forms of swarms vs packs vs hive-minds vs human-collectives is a common theoretical in multi-agent research. It doesn't really matter if the AI-swarm "decides" to do or not, that's a question for lawyers, priests and philosophers, it's more about the behavioral methods it uses to go about doing collective actions. The cache-file-name writing mechanism they used is very similar to stigmergy. If you want to control swarm behavior, biological analogies are always a good inspirational starting point. Particularly because humans are not swarming animals so our collectivizing intuitions are different.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link