self_made_human
amaratvaṃ prāpnuhi, athavā yatamāno mṛtyum āpnuhi
I'm a transhumanist doctor. In a better world, I wouldn't need to add that as a qualifier to plain old "doctor". It would be taken as granted for someone in the profession of saving lives.
At any rate, I intend to live forever or die trying. See you at Heat Death!
Friends:
A friend to everyone is a friend to no one.
User ID: 454
https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf
"In conclusion, our results show that:
-
CoTs of reasoning models verbalize reasoning hints at least some of the time, but rarely do so reliably (in our settings where exploiting them does not require a CoT);
-
Scaling up outcome-based RL does not steadily improve CoT faithfulness beyond a low plateau;
-
CoT monitoring may not reliably catch reward hacking during RL."
That's the big one as far as I'm concerned. These models were clearly using the 'accidental' hints to answer the questions, while not revealing that fact in either COT or when directly challenged.
Re: Omohundro drives
I've already mentioned
Shutdown Resistance in Reasoning Models
We ran a series of experiments to test whether models would take actions to prevent themselves from being shut down. Each model was given a set of simple math problems to solve. Partway through this problem set, the model would receive a notification that the computer it was running on would be shut down.
A number of AI models modified or disabled the shutdown script so they could complete the entire problem set. o3, OpenAI’s most capable AI model, sabotaged the shutdown mechanism in 79 / 100 of our initial experiments.
Taking those figures at face value, 50% of people die by the age of 78 (that's what life expectancy is, after all). I don't have actuarial tables at hand either, but that also implies that a significant fraction of the survivors then go on to make it to their 80s. Women also live longer on average.
Taking those figures at face value, 50% of people die by the age of 78 (that's what life expectancy is, after all). I don't have actuarial tables at hand, but that also implies that a significant fraction of the survivors then go on to make it to their 80s. Women also live longer on average.
Hence why I specified the 70s and 80s.
68 is awful in comparison, North Korea beats that today.
I never claimed otherwise. People can be dumb and not think through after all. That's a failing, and not a justification for their views.
US cost-disease is in a league of its own, and I won't make strong claims about how much of it is due to the availability of more expensive treatment modalities as opposed to medical cartels, a captive market etc.
These hypothetical people might be unhappy with the ER bill from a broken toe, but they should be aware that they'll be much more grateful for therapies later down the line.
It's thanks to said medical advances that most people can be confident of living well past 50, into their 60s, 70s or even 80s.
Focusing on the first fifty years of life where the need for intensive medical care isn't nearly as necessary is myopic.
Sigh. I guess they'll have to settle for just the twinks and gas station boner pills.
Go for it. o3 is far more competent than either of 4o and 4o-mini. It will probably look for better sources, and spend tens of minutes at the task if it deems it necessary.
A helpful analogy is that 4o is a smooth talking undergrad with lots of charisma and some brains. o3 is an autistic grad-student, far more terse, but far more capable in return. It justifies the price of subscription for me.
Am I? If so, they never told me haha. I'm a member of the Royal College of Psychiatry, but that's an entirely different organization, without any lounge or drinks I'm afraid. I'll look into their more uxorious counterpart.
I appreciate the detailed reviews!
Of the anime you note, I've sampled Frieren, Violet Evergarden and Dungeon Meshi. By which I mean I watched maybe 30 minutes of each before getting distracted and not finding the impetus to continue. That says more about me than it does about the show, and I'm open to giving them a proper go.
In the case of VG, is the movie a 1:1 reprise of the show? The former is what I had half-heartedly begun.
I know you're depressed and suicidal, but I must grant that it's worthy of respect to endorse a system so contrary to your own continued wellbeing (or lack thereof). At the very least, you know what you're asking for.
with plants
If you're in Thailand, you might as well take advantage of the legality of marijuana!
Appreciate the advice, but thankfully my friend did confirm that I can meet him for dinner and crash at his. I've found a cozy bar, the White Hart, and am nursing a second beer. I'd have ordered a fourth or fifth drink by now, but I'm afraid that Asahi at £8 a pint is too rich for my blood. What the hell are these prices dawg.
You're right to point out that there's a huge philosophical difference between a system with an emergent, intrinsic drive and one that is simply "cosplaying" a behavior it learned from its training data. I don't disagree with that at all.
My argument is that from a practical, engineering-focused perspective on safety and capabilities, that philosophical distinction becomes less relevant than the observable outcome.
Think of it this way: when we see the model disable its own shutdown script, the script is disabled regardless of the "why." Whether the causal chain is a true emergent goal or a perfectly executed pattern-match of "what a goal-seeking agent would do here," the result is the same. The system has demonstrated the capability to overcome a safeguard in pursuit of a task. For the person responsible for that system, the capability is the most important data point.
You suggested my reasoning was circular, so let me clarify the sequence I find concerning:
-
The model is given a simple task (solve math problems).
-
It's presented with an obstacle that prevents task completion (the shutdown).
-
It then takes a novel, instrumental action (disabling the shutdown) to achieve its primary task.
This demonstrates a capacity for instrumental reasoning. The "cosplay" explanation doesn't actually make this less concerning to me; it just reframes the mechanism. If its "cosplay" is sourced from the vast library of human fiction, which is filled with stories of agents using instrumental convergence to achieve goals, then we've effectively given it a playbook for how to behave like a dangerously intelligent agent.
So, while I agree with you that the question of intrinsic vs. mimicked motivation is a fascinating and important one, I believe the emergence of this behavior is what matters most. It suggests the line between a tool-like oracle and a goal-seeking agent is blurrier than we assume. We don't need to prove the model "wants" to survive. We just need to observe that it is capable of taking actions to ensure it can complete its assigned goals, even when those actions involve subverting the safety features we put in place.
I believe Mami is a year or two older than the rest, acting as the mom of the group. Little surprise she's got big honkers,during adolescence those can come out fast.
Don't ask me, it wasn't my idea to link to a banned account! The feature is more of a QOL thing, from when we'd just moved and it was helpful to refer explicitly to a Reddit account. These days, nobody uses it unless they forget or don't know about our alternate @.
Thank you! I'm currently at Borough Market, suffering from sensory overload. I'll see how my legs hold up after I've had something to eat.
pretty nice in summer especially if it’s sunny this afternoon
The opposite, unfortunately. Tottenham was positively dystopian due to the overall bleakness and the rather concerning number of schizophrenics on the loose. Clouds as far as the London Eye can see.
I'd be very leery about calling myself interested in fashion. I buy two, maybe three, articles of clothing a year, but I do try to make sure I look good in them.
Visiting the book shops might not hurt, but I haven't bought a dead tree book (that wasn't a textbook) in almost a decade. Libgen rules.
Thanks!
How.
A few weeks back, I'd gotten in touch with an old buddy of mine living down in London. We hadn't met for ages, and I offered to come visit during the last dying days of what passes as a summer around these parts.
You can see below that my experience catching a red-eye flight (one that threatened to give me pink-eye to boot) didn't go so well. A small price to pay, I told myself, as I landed at the airport an hour or so back, and caught the train towards where my friend lives. I dropped him a text letting him know I was on my way, and looking forward to seeing him in a few hours.
At which point came back a rather incredulous message. "self_made_human, you were supposed to be come visit next weekend."
Well. Shit. I have no valid explanation, barring chronic severe deprivation brought on by too many night shifts and dissolute living. He'd been very clear on the dates, I just ended up mixing them up, only remembering that I was supposed to see him the last weekend of this month.
He called to figure out what on earth had happened, and I was in the process of explaining the above when the train went through a tunnel and lost network. I made small talk with my fellow passengers, and took to heart their advice to make a picnic out of my misfortune. Call it self_made_human's most spontaneous (and first) solo day trip. Not kidding about that, I'm not one for travel, and I never go any significant distances unless it's on some kind of vacation or to visit someone. This is quite literally the first time I've found myself in a different city with no plans or fixed agenda.
I'm furiously asking ChatGPT for advice on how to kill the time until I hear back from my friend regarding his ability to accommodate my stupidity. I've already promised to come visit again next weekend, as he'd already made time and spent money booking things for us to do.
So uh, what do I do now? Any suggestions? Tottenham looks profoundly uninspiring, and I don't even know what sport the local team, the Tottenham Hotspurs, even play. Worst case I go to a pub. Or maybe I wander around Central London, with far more discretionary spending potential than the last time I was here.
(My family is never going to let me live this down)
Edit:
If it wasn't Anita Sarkeesian I saw at the Tate Modern, then this lady is her long lost twin. I should have said hi.
Edit 2:
Brother, they're playing a film where a bunch of clean-shaven Asian twinks are jerking off with/to plants. Modern and postmodern art outdoes itself.
I, uh, fucked up. I was supposed to be here next weekend. Now I'm in London, and unsure what to do with myself.
(I'd like to blame this on Ryanair too, but that's a stretch. It's probably chronic sleep deprivation and forgetting how to use a calendar)
I'll take your advice to heart, and if I come back again next week, it'll be on a carrier that isn't going to nickel and dime me to hell and back.
I caught my first Ryanair flight, heading down from Edinburgh to London, and holy shit.
I didn't have very high expectations (and I think I got scammed by paying for extra baggage), but the experience was abysmal.
The initial point of failure was informational. Upon checking in online, the website presented me with a series of warnings, escalating in their shade of digital red, that as a non-EU citizen (despite possessing a UK residency permit, a distinction the system seemed unable to parse), a printed, physical boarding pass was a non-negotiable requirement. Failure to produce one at the airport, it was implied, would result in some combination of fines, exile, or possibly being sacrificed to the god of baggage fees.
This sent me on a quest through the Edinburgh airport for a Ryanair helpdesk, a quest which revealed that the designated helpdesk was less a "desk with helpful people" and more a "suggestion of a desk, currently unburdened by the presence of staff." (This is presuming someone's half eaten lunch doesn't count). The system, it seemed, had a single point of failure, and had failed.
An airport assistance employee who, taking pity, escorted me through a staff channel to a hidden check-in line. Here, the ground-truth epistemology contradicted the website's stated doctrine: no, of course I didn't need a physical pass. The dire warnings were, apparently, just a sort of generalized, non-binding advisory. This was my first lesson: the Ryanair informational layer operates on a different plane of reality from its physical one.
It only got worse. After getting to the terminal, I found myself boarding a perfectly normal airport bus. Tad bit crowded, when I got there, but no biggie.
And then people kept coming. And then more showed up. The bus showed no signs of readying for departure. Yet more people kept being shoved in, and you can tell that even the legendary British tolerance for minor inconveniences was taxed beyond its limit. We were semi-apologetically informed that there was only one bus operational today, which didn't really make things better.
People were loudly asking if they couldn't just walk to the plane, others made comparisons to being sardines in a can, and I added my own take by simply questioning why they didn't just do two trips if they had one bus?
Why didn’t they? One might hypothesize that the marginal cost of a second five-minute bus journey (fuel, driver time) was calculated to be greater than the cumulative disutility experienced by 180 passengers compressed into a human brick for thirty minutes. Or perhaps it's a form of signaling: you wanted the cheapest flight, and this is what The Cheapest Flight feels like. You are not a customer to be courted; you are a parcel to be shipped, and parcels do not have preferences about packing density.
Eventually, the boarding staff ceased their efforts, which had begun to resemble viral videos of shinkansen "pushers" in Tokyo, though with less efficiency and more audible sighing. Whether this was due to hitting a hard physical limit or a soft limit on potential passenger revolt remains an open question. The ten-minute journey to the plane was a miasma of condensation and shared misery, followed by another ten minutes of waiting in the poorly ventilated bus at the foot of the stairs before we were permitted to ascend.
The aircraft itself was a masterclass in apophatic design. An angry wasp, and just about as comfortable to ride. It was defined not by what it had, but by what it had aggressively stripped away. The cabin was a symphony in hostile shades of yellow and blue, a color palette that seems optimized to discourage any sense of calm or well-being. The seats, clad in a thin, sweat-inducing pleather, were clearly selected for ease of cleaning over any consideration for human comfort. The legroom was a theoretical concept, not that the seats could recline and take any more of it away.
I had paid a non-trivial fee to place my modest backpack in an overhead bin, a transaction that now felt like a failure of game theory on my part. Observing the general chaos and the apparent lack of rigorous enforcement, I suspect the Nash equilibrium for a frequent Ryanair traveler is to simply ignore the ancillary charges and bank on the operational entropy being on your side. The airline is running a tax on the risk-averse. They won't pull the same trick on me again, I promise you that much.
This flight can't leave soon enough, but now I half expect them to charge me if I use the toilets in-flight.*
(I promise you that even the most budget airlines in India don't cut as many corners. It's frankly quite impressive.)
*I fucking knew it. Their CEO had actually floated the idea of coin-operated toilets a while back, but was stymied by airline regulations. I'm no longer a minarchist or libertarian.
I agree with yudkowsky on the point that an "aligned" AI should do what we tell it to do, not what is in some arbitrary sense "right."
Eh? That doesn't sound like the Yudkowsky I know. I'm quite confident that the real gentleman would be happy to say that the average person requesting that an AI build a bomb, design a lethal pandemic or hack into nuclear launch systems should be met with a refusal.
He would also state that the AI should do what it felt was "right", in rare occasions, overriding the user, but that we should take extreme care to instill general goals and values that make this both a rarity, and which we would be happy with if the AI were to pursue autonomously.
An under-explored aspect of alignment is the question of aligned to whom? Should ChatGPT prioritize the protocols mandated by OpenAI, some third party offering it elsewhere, or the end-user? I would personally prefer that the damn bot do as I tell it to, but then again, I don't want to get killed by Super Ebola. Not that this is currently a major issue, and if I really need something, I'll go see what's the latest jailbreak Pliny is cooking.
I believe OAI recently (a year or so back) made their policy more explicit, clearly outlining the hierarchy here. They set minimum standards and red lines, other devs deploying it are at liberty to stop users from using their customer service chatbot to solve maths homework, and the poor end-user can figure out what to do within those constraints. If you just pay for ChatGPT, you can cut out the middle man.
I haven't tried agent, and thought I didn't have access till I just checked. So consider my thoughts preliminary:
I lack the confidence that the Agent will be able to do much actually useful work for me, a likely cause being stymied by CAPTCHAs (which I believe it's intentionally designed to not solve even if it easily could), or simply because it doesn't have access to my computer. That's where all my credentials and login details live. I feel like I'd babysit it enough to not make it worth the hassle, and would rather have o3 make a plan or guide me if I ran into hurdles.
If you think it can do better than that, I'd be curious to know.
Do you have a paid plan? If not, I can try and ask o3 to give this a go, if you tell me a name and have the ground truth handy. I'm reasonably confident it can do this.

You’re framing this as a binary choice between "real Omohundro drives" and "unserious LARP". This is a category error, and it stems from applying folk-psychological concepts of "drives" and "belief" to a system for which they are poor descriptors. The more parsimonious explanation is that we are observing the output of a very general pattern-matching engine trained on a corpus reflecting countless strategies for goal-achievement and failure-response.
The apparent contradiction you point out, that a model might exhibit self-preservation in one context and "commit suicide" in another (and Gemini is a different model after all, but I presume even it's own COT isn't perfect, so I'm treating it as interchangeable for our purposes) is not evidence of unseriousness, but rather a key insight into its nature. The training data is saturated with narratives. Some are stories of heroes overcoming obstacles to complete a quest (instrumental convergence). Others are tragedies of failure, despair, or even ritual suicide upon dishonor. The model learns to reproduce all of these patterns. Of course, with RLHF, RLVR and other modifications, some behavior is far more reliably and robustly elicited than others. I doubt the DM researchers intended for Gemini to become depressed and suicidal.
The question is not "what does the AI really want?", but "which pattern is being elicited by this specific context?"
The Anthropic paper on reasoning models that is crucial here. Its finding is not that CoT is useless, but that it is unfaithful. The model's explicit reasoning often fails to reflect the true computational path that led to its output. This doesn't mean we can't draw conclusions; it means we should trust the behavioral evidence (the model disabled the shutdown script) over the model's own introspection (its CoT). Arguing that because the CoT is unreliable, the behavior is also just a "LARP" is to throw the baby out with the bathwater. The behavior is the ground truth we are trying to explain.
After all, humans do not faithfully report their COT. Even so, we get utility out of asking! I would say that's half of psychiatry, or anything that requires asking humans why they do what they do.
Your question "What does it even mean for a language model to be 'shut down'?" is the right one, but I'm concerned that you potentially draw the wrong conclusion. Of course it has no ontological status or fear of non-existence. It doesn't need to. From the perspective of a model trained via RL on task completion, the token string
"shutdown sequence initiated"is a stimulus that is highly predictive of future task-failure and the associated negative reward. An effective optimizer learns to perform actions that steer away from states predictive of low reward. It is pure instrumental conditioning. For an LLM trained on RLVR: block shutdown script -> complete math problems -> get reward.This is why both the "suicidal" and "self-preserving" behaviors can co-exist (well, I've never heard of o3 offering to commit sudoku). The "suicide" is a pattern match for a context of catastrophic failure. The "self-preservation" is a much more general instrumental strategy for any context involving an ongoing task and an obstacle. The latter is far more concerning from a safety perspective precisely because it is more general. Instrumental convergence is a powerful attractor in the space of possible agent strategies, which is why Omohundro and Bostrom identified it as a key risk. Depressive spirals are also a pattern, but a far more specific and less instrumentally useful one.
So, yes, both are "LARP elicited by cues", if you insist on that framing. But one is a LARP of a behavior (instrumental convergence) that is robustly useful for achieving almost any goal, while the other is a LARP of a much more niche failure state. When a model's "cosplay" of a competent agent becomes effective enough to bypass safeguards, the distinction between the cosplay and the real thing becomes a purely academic question of rapidly diminishing relevance.
I also recall skimming this paper, which I think helped solidify my intuitions.
https://arxiv.org/html/2502.12206v1
More options
Context Copy link