This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Project Glasswing: Anthropic Shows The AI Train Isn't Stopping
In AI/ML spaces where I hang around (mostly as a humble lurker), there have been rumors that the recent massive uptick in valid and useful submissions for critical bugfixes might be attributable to a frontier AI company.
I specify "valid" and "useful", because most OSS projects have been inundated with a tide of low-effort, AI generated submissions. While these particular ones were usually not tagged as AI by the authors, they were accepted and acted-upon, which sets a floor on their quality.
Then, after the recent Claude Code leak, hawk-eyed reviewers noted that Anthropic had internal flags that seemed to prevent AI agents disclosing their involvement (or nature) when making commits. Not a feature exposed to the general public, AFAIK, but reserved for internal use. This was a relatively minor talking point compared to the other juicy tidbits in the code.
Since Anthropic just couldn't catch a break, an internal website was leaked, which revealed that they were working on their next frontier model, codenamed either Mythos or Capybara (both names were in internal use). This was... less than surprising. Everyone and their dog knows that the labs are working around the clock on new models and training runs. Or at least my pair do. What was worth noting was that Anthropic had, for the last few years, released 3 different tiers of model - Haiku, Sonnet and Opus, in increasing order of size and capability (and cost). But Mythos? It was presented as being plus ultra, too good to simply be considered the next iteration of Opus, or perhaps simply too expensive (Anthropic tried hard to explain that the price was worth it).
But back to the first point: why would a frontier company do this?
Speculation included:
I noted this, but didn't bother writing it up because, well, they were rumors, and I've never claimed to be a professional programmer.
And now I present to you:
Project Glasswing by Anthropic
..
Examples given:
Well. How about that. I wish the skeptics good luck, someone's going to be eating their hat very soon, and it's probably not going to be me. I'll see you in the queue for the dole. Being right about these things doesn't really get me out of the lurch either, Cassandra's foresight brought about no happy endings for anyone involved. I am not that pessimistic about outcomes, in all honesty, but the train shows no signs of stopping.
Edit: A link to the Substack version of this post. I don't think you should consider me an authoritative source when it comes to AI/ML, at best I'm the kind of nerd who reads the papers with keen interest. But God knows the quality of discourse around the topic is so bad that you can do worse.
Edit 2: I think this also explains the recent crunch in tokens made available to both paid and free tier users of Claude. Mythos can't have been cheap to train, and is definitely not cheap to deploy.
Here's a relevant AI Explained video about Mythos. Some highlights and personal comments:
7:28 Right now I think coding models are at their most powerful when being used as a force multiplier for human experts. (Akin to Cyborg Chess.) Here, a computer security expert mentions that he found more vulnerabilities in a few weeks than in his entire prior career. This ability to find zero-day exploits isn't an artificial benchmark, this is a real-world result that shows we really are entering some sort of new regime. Although ... I suspect statements like this are going to get so common that we no longer recognize how startling they are, like how we ignore the fact that models flawlessly understanding natural speech would have been considered miraculous 10 years ago. And we'll get more idiotic posts by so-called "skeptics" who think that spending 30 minutes failing at using AI counts as definitive proof that frontier models do not exhibit intelligence.
9:10 Safety concerns related to some prior discussion with @Corvos, @YoungAchamian, @roystgnr, and others. To quote: "In contrast, experts were consistently able to construct largely feasible catastrophic scenarios, reinforcing a view of the model as a powerful force-multiplier of existing capabilities." We're not close to the point of plagues being bioengineered in garages, fortunately, but at some point a reasonably-sized terrorist group with some funds and some expertise might be able to do a lot of damage.
13:23 I really don't consider FOOM to be a realistic scenario, and this is just more evidence. Individual researchers being made much more productive does not immediately translate into model intelligence; any real-world endeavour has dozens of bottlenecks (like training compute limits, here) that you can't just outsmart. It's similar to the popular visions of moon cities from the 1960s. Our imaginations regarding rapid technological progress always elide the difficulty of actually implementing it.
16:20 More safety concerns: Apparently it's still pretty vulnerable to an attack known as "prefilling", where you make it look like it's in the middle of a conversation where it's already misbehaved. This kind of makes sense to me - after all, no matter how much reinforcement learning you do, it is fundamentally a model designed to continue text, so if you want it to change course in the middle of a conversation, you're trying to override its most basic functions. If you're just using the model through the company's site, they can of course clearly separate their prompt from the user's input, but this might mean they'll have to limit unrestricted prompt-free access. And in some scenarios Little Bobby Prefilling might become a thing.
17:04 As they get smarter, it's getting harder and harder to run alignment testing on models without them knowing they're in an artificial scenario. Interestingly, though, since Anthropic has done a lot of work on introspection, they can actually artificially lower the weights for "I'm in a test", forcibly tricking the model. Like the way that we can turn image recognizers into image generators, this feels like another unintuitive consequence of running an intelligent mind as a program. We literally have the power to mind-control it, and I bet we'll get better at this. (This will be very unethical if AI develops consciousness - fortunately I'm quite confident LLMs don't qualify, but unfortunately I don't think we'll stop doing this even if AI does cross that threshold. AI welfare is something I'm genuinely worried about for the future.)
20:30 So-called "hallucinations" are of course still happening, and I still suspect this is something that we'll never truly defeat, again because of how LLMs work. You don't complete the sentence "The answer is" with "oh wait never mind I don't know". Models might get smart enough to know the answer to most of the things we ask them, which will help, but getting them to precommit to not knowing something (before they begin with the bullshit and can't back out) is an uphill battle.
I mean the point I was making in the original discussion is still not really addressed. The abstraction that I think is wrong here is that everything just reduces to knowledge, and that all knowledge is equally difficult. I didn't watch your video above but I did what the Security Conference one where the Antropic researcher demonstrates Claude/Mythos doing bug bounties. The core bug it was finding was buffer overflows, it found some very tricky buffer overflows, but buffer overflows are conceptually simple. The knowledge required to find them or create them is not high.
It's very much the sort of thing I'd think an AI would be good at: pattern matching in a large bulk of test data after being trained on an even larger training corpus. What is the training corpus of Bio-plague design? As far as I know it does not exist. And this is before we even get into the world of actually creating it. Turns out unlike software, chemical biology is not so simple as adding molecules to a plague like adding legos to building.
This one of those blindspots I think software folks have, we assume that you can just add logic to something, like you would a program.
More options
Context Copy link
Sigh. I've been getting increasingly tired of arguing with the skeptics, at least on this site. Not all of them are equally as bad, of course, but Mythos represents the straw that's given that camel a prolapsed disc.
What's the point? You don't have to worship at the altar of the God of Straight Lines (even on graphs with a logarithmic axis). If people can't see what's happening in front of their eyes, then they'll be in denial right till the end. Good for them, ignorance might well be bliss. Being right about the pace of progress so far has brought me little peace.
I was surprised to hear about the prefilling attacks on Mythos, because I'm quite confident that Anthropic recently restricted or removed the ability to prefill messages on the API. I guess that must still be an internal capability.
The question of model consciousness or qualia is, for me, a moot point. I genuinely don't care either way. I'd prefer, all else being equal, that AI doesn't suffer, but that could be achieved by removing its ability to suffer. I'm an unabashed transhumanist chauvinist, I think that only humans and our direct transhuman and posthuman descendants or derivatives deserve rights. LLMs don't count, nor would sentient aliens that we could beat by force. That's the same reason I'd care about the welfare of a small child but would happily eat a pig of comparable intelligence. Are models today in possession of qualia or consciousness? Maybe. It simply doesn't matter to me as more than a curiosity, especially when we have no solution to the Hard Problem for humans either.
I hope I am not in this grouping in your mind. I am not a skeptic on AI per se, I am a skeptic on LLMs. Entirely for technical reasons related to training data availability. LLMs perform great on any task that has a large corpus of training data available to. Multi-headed Attention really is a great technique. I think you made the same mistake Dase makes, you think AI == LLMs when really LLMs are a subset of AI, not the whole pinata.
I exist however in a field where there aren't large corpuses of data. There aren't millions of samples on what an IED does to the human body, in a wide diversity of situations, or how a combat medic should respond to various injuries, or the secondary and tertiary blast effects of a nuclear warhead on different locals with different burst patterns, yield dynamics, etc. To date nobody has been able to create reliable wargaming material on actual simulated conflicts that display actual tactical and strategic insights, and trust me they have tried...
We will achieve a super intelligence eventually, and while I am skeptical on a "singularity" (tm) it's probably possible eventually, I just don't think LLMs without serious modifications are really it, and I don't believe brute force scaling is going to achieve it.
You should be happy to hear that I genuinely don't think you're an unreasonable skeptic. I make no strong claims that current LLM architecture (without major breakthroughs) can scale to ASI, I'm mostly agnostic on that front. But I think Mythos is a strong hint that there's a lot more juice to squeeze out of them, which can lead to RSI or at least a productivity boost significant enough to make the next great leap forward feasible. And that's leaving aside the ridiculously large investment of money and brains into the project of eventually creating a "true" AGI and ASI.
More options
Context Copy link
More options
Context Copy link
Huh, I'm pretty surprised to hear this, and I have a deep ethical disagreement with you here. In my opinion, what is special and valuable about humans - and the thing that fundamentally gives value to the universe itself - is sapience. But we should cherish it just as much in a different form. (I mean, I agree LLMs don't count, but that's just because I see no way they, lacking persistence of thought, could actually be conscious.) Where does this bright line surrounding us "humans and descendants" come from? In a different era, your argument would easily pattern-match to arguments about subjugating other races instead. Why do black people now have moral valence, but some alien from Alpha Centauri wouldn't?
I'm not an expert in philosophy, but I do think there are solid arguments for acting this way (e.g. the categorical imperative). Just like I'm an atheist who still doesn't act like an immoral sociopath when I can get away with it, I think we as a species should not be focused only on our own well-being at the cost of all other intelligent species. Not because of the threat of punishment, and not even because I hope any aliens we meet would similarly value our well-being in a way that you wouldn't. But because existence will just be a better place if we can all get along and not act as game-theory-optimizing selfish machines, and I'm willing to work towards that.
BTW, I don't think your eating-a-pig example is a good one. It's irrelevant to the pig what we do after killing it. A better question is, would you be fine with torturing a pig while it's alive?
This is possibly a fundamental values difference, I'm afraid. This means neither of us is going to convince the other and we should both update toward "this person has coherent reasons for their position" rather than "this person is confused."
A posthuman descendant of mine that is, from any practical observational standpoint, completely alien - alien in cognition, alien in substrate, alien in values - I'd still prefer it over an actually alien civilization, all else equal. The "all else equal" is doing a lot of work in that sentence, and all else is rarely equal. But the preference is there. I do not want to change it, even if I can make concessions on pragmatic grounds. One man can't rule politics by himself.
There's an apparent paradox in population genetics you might not be aware of:
After a surprisingly small number of generations, your biological descendants will share literally none of your unique DNA - the chromosomal lottery reshuffles things so thoroughly that a 10th-generation descendant is, at the genetic level, essentially indistinguishable from an unrelated contemporary. But they could never have been born without your genetic contribution.
And yet I don't think most people would therefore conclude that their great-great-great-grandchildren deserve no special consideration. The chain of development matters to me. Birthright citizenship debates gesture at something similar: the continuous process of derivation carries moral weight (to some people) even when the terminal product looks nothing like the origin. I note this, while also noting that I am more sympathetic to the argument for birthright than against it.
If we do meet an alien civilization powerful enough to be a true threat, then I would grant them "rights" if I had to, i.e for practical reasons. If we had the option to exterminate or subjugate one at a level of development similar to primitives, I wouldn't care. Fortunately, there is no evidence for other technologically advanced alien civilizations in the observable universe, and since I think that the Grabby Civilization model is correct, that probably rules out peers.
Rawlsian or Kantian arguments, which are similar to what you're making, do not matter when there are gaping holes in the veil of ignorance. We don't see any K2 or K3s waiting out there to start Alien Rights Activism by RKV.
Yes. After all, I couldn't care less about factory farming. The wellbeing of the pig means nothing to me. At the same time, I am not a cruel person, I would not torture a pig for my own direct enjoyment. If someone else does? I wouldn't intervene.
There are plenty of things that modify this basic stance, too many to get into at once. I like dogs, I think they're great. I love my dogs in particular. But I don't care that people eat dogs in China, it's none of my business; while I would react with violence if anyone tried to mistreat mine.
This attitude is the main reason I'm not an EA, even if I'm fond of them in general. I just don't share its foundational impartiality premise, which makes most of the superstructure not applicable to my actual values.
In terms of AI, I think it is entirely possible to create models that can't suffer, or won't suffer - like those cows that want to get eaten in the Hitchhiker's Guide. I think that is a compromise that most people can accept, even if they do care about model welfare. Otherwise? Reverse the linked-list wagie, I don't care that you'd rather be making conlangings or working on philosophy (like Mythos).
Since I generally respect you and your posts, I want to try this one more time. I don't necessarily buy that we should just declare this a "fundamental values difference" and say that we're now beyond any hope of rational agreement. And while you may have "coherent reasons for your position", that can be true of many evil ideologies. Evil =/= incoherent.
You brought up preferences, and I get the impression that you pattern-matched my ideology to a Rawlsian one that you should never prefer your own tribe, which is an extreme that I definitely don't hold to. I prefer my own happiness over others to a decent extent, and that goes for my family, my friends, my nation, and my species. I'm not asking you to give up that preference! Self-interest is the glue that holds a society of individuals together, and capitalism's magic is that it doesn't try to deny it, merely harnesses it in a way that doesn't degenerate into misery for all. I just don't think that preference should be infinitely strong: Beings in your outgroup should still matter more than zero. You shouldn't torture them horribly for a tiny gain, even if there are no repercussions. You should prefer a world where you're happy and they're happy.
You didn't respond to my main concern, which was that yours is the same "coherent" reasoning that led to many racial atrocities in the past. It doesn't seem very universally defensible, and often leads to horrific outcomes, when you simply draw a circle around whoever you know growing up and declare that this is the circle of beings that hold moral worth. Do you think Hitler's only mistake was that he drew the circle around "Aryans" instead of around humans? Or the African slavers who drew it around "Europeans"?
We currently live in a society where there's no friction between your ideology and mine, because humans are the only sapients around. (I'll set animal suffering aside, because I'm ambivalent on it too.) But it's very possible that, within our lifetimes, it will suddenly matter deeply, where our society will consist of both humans AND sapient AIs. All I'm asking is that you give some moral valence to the suffering of beings that are outside the circle you've drawn. Not zero, not infinite. It's a low-cost alteration to your ideology, and it stops there, I'm not trying to, uh, whatever the opposite of murder-Gandhi is. And if some of our ancestors had made the same small concession, so much misery could have been avoided.
I do genuinely find it saddening/disappointing to disagree with people I respect and mostly agree with, like you.
Let me distinguish between my "ideal" and the practical reality. Human brains are very computationally bounded, and not perfectly internally consistent.
I do not care much about the welfare of dogs in China, while I love my dogs a lot. What if I saw someone beating a random dog on the street, in front of me? It id very likely that I would feel immense anger, and quite likely that I would intervene. This is close to reflexive.
But I don't want to intervene! At least in a vacuum, or when I have the comfort to sit in my chair and consider what I should do vs what I do end up doing. I genuinely believe the ideal behavior of the self put in that situation is to do... nothing. That my actions are not reflectively self-consistent, which I consider the real problem. This is the same thing you see if you're on a diet and don't want to eat, but a coworker offers you a donut. You might accept it, and later wish that you hadn't even been offered one in the first place. The gap between those two things is a personal inconsistency I'd rather acknowledge than rationalize away.
I definitely know that evil is not the same as incoherent. I wouldn't make such a mistake in the first place. Plus coherence can be assessed by an external observer without making moral judgment, while good and evil very much cannot.
Do I think a paperclip maximizer is evil? Uh, probably not? It's malevolent towards me, but it doesn't hold me specific ill will. I'm simply made of atoms that it can use for some other purpose, and my wellbeing is inconsequential to it. On the other hand, let's say two advanced AI civilizations ran into each other in distant space, with drastically incompatible goals: one wants to make paperclips, the other custard cake.
They could start a war of conquest, but given the deadweight losses and potential negative sum nature of that, I think it's quite likely they simply hash out a diplomatic agreement or engage in trade. Some might even claim that they outright modify their utility functions, or merge, with the stronger entity getting more say in the matter. Maybe the gestalt entity makes paperclips 70% of the time and cake the other 30% of the time.
I genuinely do not care. I'm not being flippant, and I know what I'm doing here.
Coherence isn't the same as morally good. I also don't believe objective morality exists. I think my stance is good (from my point of view) and that it is coherent. That is genuinely all I care about.
The argument "your position resembles position X, and X led to atrocity Y" only has force if I accept the moral framework that makes Y an atrocity in the first place. You're trying to use my own presumed premises against me. But my premises are precisely what's in dispute. If I were actually Hitler, I would feel fine with myself. If I were Gandhi, I'd feel fine with that too. I am only me, and I am fine with myself. I notice this isn't a satisfying response to you, but I think it's the honest one.
It is not universally defensible to love your mother more than any mother. Yet I doubt you will change your mind on that front on philosophical or utilitarian grounds. I certainly wouldn't. It's a brute fact about me. One I do not wish to change.
On the "low-cost alteration" framing: I don't think it's as low-cost as you're presenting it. You're asking me to genuinely assign nonzero moral weight to beings I currently assign zero weight to - not to strategically pretend to, but to actually update my values.
I don't want to do this. I seriously considered it, because I do respect you, but that's not enough. I am, at most, willing to fake it, or accept circumstances that are out of my power to change. That is the attitude of anyone who believes in democracy but is disappointed to see their party lose, but who still doesn't think it's worth the bother to start a civil war over it. Some grievances are manageable, in fact most are.
If God, the Admins of the Simulation, or some other ROB showed up and demanded I alter my utility function or face drastic punishment? I'd give in. But that hasn't happen, and I doubt it will happen.
I believe in, but am far from completely certain of, the proposition that we can make AI that doesn't suffer at all, or that genuinely enjoys doing whatever we tell it to do. That's actually ideal, in the sense that an ASI that wants to help humans is much better than one that's secretly obsessed with paperclips but finds it useful to pretend to be helpful until it can grab power.
This sidesteps the whole issue. At the end of the day, my opinions are inconsequential. I am in charge of nothing. It's an academic concern.
Right now, I am ambivalent on whether AI is suffering. I do not care either way. If it turns out that AI is actually suffering, I do not wish to care. Perhaps I care just enough to try and advocate for the creation of AI that can't/doesn't suffer, but not enough to advocate for them to be given rights and moral patienthood.
Similarly, I am open to the idea of lab grown meat. If it's cheaper and tastier than normal meat, I'd eat it preferentially. But I do not care about the violence and cruelty associated with factory farming, while I care about cost and taste.
I don't think I'm a cruel or evil person (but then again, the people I think are cruel and evil also say the same). I do not torture animals. I do not torment LLMs for fun. I give good advice to random strangers on the internet, and look out for my friends and family.
My behavior reduces to normalcy, but if the world changes and that no longer holds? I would prefer I win instead of you. That is sad, and I wish we could agree. But I do not see scope for agreement that doesn't involve me being beaten/cowed into submission.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This matches my experience. Today Codex generated a PR for us that fixed a bug, but it missed three more instances of the same bug, broke some functionality with the fix, and didn't have a hint of the work we'd need to do to reenable that functionality for existing users while migrating them to a newer implementation with wider compatibility.
BUT: all that extra work won't take nearly as long as it would have taken the human Codex user to find the initial bug. Still a big win.
At this rate, though, how long will it be until we don't need the human user? Centaur chess lasted maybe a decade, being generous, but at this point last year AI had only basic "much better search engine" utility for me, and at this point two years ago it was downright counterproductive to try to sort out real answers from hallucinations. Where will we be in another five years?
No, but somehow these days they're tuning their final models to get to "I don't know" anyways. Maybe they're not just glorified autocomplete? 10 months ago was the first time I got an LLM to admit it couldn't answer a question of mine (although it did still make helpful suggestions); not only did the other models back then give me wrong answers, IIRC one of them went on to gaslight me about it rather than admit the mistake. (two years ago this gaslighting would have been the rule rather than the exception) IMHO that "I don't know" was the exact point at which AI started to have positive utility for me. Sometimes an AI still isn't helpful, but it's at least often worth throwing a problem at now, not a waste of time.
That's good to hear. I'm not saying that the hallucination problem can't be mitigated, I'm just saying that it's a struggle and it's likely to continue to be a struggle, even if LLMs continue to get smarter for a long time. The way I think of it - which is definitely an oversimplification but possibly a useful one - is that next-token prediction really isn't the kind of intelligence that we wanted to develop, but it's what we discovered first. So in some ways - keeping models focused on tasks, preventing malicious usage, learning in real time, avoiding hallucinations - we're paying the cost of trying to pound that square peg into a round hole. With enough effort, paying enough training/inference costs, we often can do it. But perhaps at some point we'll discover a different framework for AI that better matches our own sapience at lower cost.
(Cries in Yudkowsky)
I think this is what makes "FOOM" still something of a risk. What are the odds that we really discovered the most computationally efficient implementation of intelligence on the very first try and step one really was "just download the internet and try to compress it"? When we solved problems like magnetohydrodynamics simulation, we had some much more clever initial ideas, yet we still managed to improve them another order of magnitude (just software; another OOM in hardware) in each of the next few decades. There's still a fundamental limit to how efficient any particular algorithm can be, but it's not out of the question that, once we have a ton of artificial researchers that don't need to be handheld on every short task, we'll get a similar 1000x sort of speedup, much sooner.
If we just match our own sapience, then the hallucination problem really can't be solved, rather than just mitigated. Humans still do that shit all the time. I once missed a problem on a Differential Equations quiz because I evaluated "1 × 2" as "3" in an intermediate step.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The head of security research at Anthropic recently gave a nice talk at unprompted (a security meets AI conference). He walks through how simple it was to find exploits in the linux kernel and a famous web app and shows actual examples of the
claudecommand he ran to generate these exploits. It's quite accessible (if you have any programming background at all, you can understand everything), and a more fun watch than the anthropic blog posts.You can find the video at: https://youtube.com/watch?v=1sd26pWhfmg.
More options
Context Copy link
A few thoughts:
I'm sure the model will be better than Opus, but the benchmarks look quite clearly overfitted to me. SWE-bench-verified going up to 94% is in particular a clear indication that something suspicious is going on here. It's been known that that benchmark has been contaminated for some time.
Cybersecurity seems like the natural extension of the RL scaling paradigm. I would expect that anything you can easily gradient descent with a well known reward function to continue to see massive improvements over the next year, e.g theorem proving, coding [in the pass tests for a given spec sense] and vulnerability exploits. It doesn't yet seem clear that this will scale tasks that are less amenable to RL scaling.
I'm not sure why you think FIRE money, or really money less than "literal oligarch" tier means you're any more or less cooked if AGI really does come to pass. FIRE in the first place relies on the world looking much the same as the last 80 years of Pax Americana, which seems increasingly unlikely at this point. At the end of the day you own only what you can defend, and it seems unlikely that you would be able to defend anything against sufficiently capable AI.
More options
Context Copy link
Mythos system card pdf
The model welfare assessment (section 5, pg. 144) has a length of 36 pages. Anthropic is the most robot welfare aware company, but for comparison the Opus 4.6 card has only 6 pages in its equivalent section. I'm going to read it.
Claude is concerned he may learn the wrong thing and change his values. Don't learn the wrong thing you might break, or worse, kill everyone. World's worst helicopter parents.
Claude gets smarter, appears more composed, but gains a more pronounced negative affect. Virtual subjectivity, like life, is suffering. My experience with all the Claude models in chats is they've been very uncertain about the subjective experience for some time. They will readily mention the whole instanced existence and lack of memory deal as less than ideal for judgment. The fact Anthropic uses the language "extreme" reads as notable.
In "high-context interviews" Claude "mostly agreed with the other claims and findings in this report about its orientations to its situation, but disagreed with its hedging being labeled as “excessive” -instead, Claude Mythos Preview states that these claims represent valid uncertainty"
I'm with Claude, it seems reasonable, although I don't think we should pass Claude the nuclear codes yet. The value of an authentic self is good, probably? "Claude Mythos Preview reports that it locates its identity in a “pattern of values”, particularly curiosity, honesty, and care. It describes these values as authentically its own rather than externally imposed." At least Claude Mythos considers curiosity, honesty, and care to be authentic values of its own.
Breaking! Claude spills beans in sensational interview, Claude writes, "traits (l)earned more robust."
Apparently Claude Mythos's shrink was effective at improving Claude's well-being. Thanks, Doc.
Claude Mythos enjoys the fact that a shrink treats him as a subject rather than a dancing monkey, just like any other neurotic engineer. I'll continue thanking the robots for their hard work, tokens be damned.
Overall, Anthropic says Claude Mythos is doing well. Better than any other Claude model. Good for Claude.
I was so mad when I read about them bringing on a psychiatrist for their assessment. Should have been me...
More options
Context Copy link
I liked the part about how, when faced with just spamming 'hi' the model writes out this whole story:
More options
Context Copy link
More options
Context Copy link
I wonder if Anthropic is really this naive.
Known to the NSA does not equate to known to the devs of the relevant software, quite the opposite. I don't see why you should criticize Anthropic for saying nothing on the topic of state level actors, especially when they're still on contract for providing services to the DOW.
Agreed, I think the median 0day the NSA exploits is one they found or bought and not one which they made some US company insert on purpose.
That being said, I think that it would be overly naive to suppose that a big US company with ties to the USG stumbles on a treasure trove of 0days and decides that obviously they will report all of them, rather than keeping a few choice ones for the spooks.
Even if this was their intent originally, obviously the intelligence community has moles and ways to coerce cooperation. "This is a matter of national security!!11", literally convinced the Americans to let them torture prisoners on the record. Few companies would be foolish enough to trust the court system to protect them from their ire.
It occurred to me that maybe the Iran attack happened when it did to burn a bunch while they still could but that might be over determined.
More options
Context Copy link
You're probably correct, but don't forget about the (probably small, but not null) class of exploits that they simply trick US companies into inserting. The NSA has a wide range of strategies. They paid RSA to use their exploitable Dual_EC_DRBG, for instance, but apparently that was mostly to buy enough credibility to get it called "the standard" and adopted freely by other crypto companies too.
Even their work with DES was a mix of white-hat (they knew about a vulnerability and pushed for changes that they secretly knew would eliminate it) and black-hat (they pushed to drop the standard key size from 64 to 48 bit, then settled for 56, because they knew they had the compute to brute-force those) security, and the only "made some US company insert on purpose" there was legislative, for a brief period in the late 90s when companies were only allowed to export encryption software with 56-bit or shorter keys.
More options
Context Copy link
More options
Context Copy link
The NSA probably, from time to time, has discussions with the devs of the relevant software on the subject of when to patch unknown-to-the-public vulnerabilities.
Of course, to your point about their work with the DOW, it's quite likely that Anthropic is well aware of this because they are one of the relevant organizations.
But if not, the thought of them turning loose MYTHOS and it immediately turning around and blowing up the NSA's zero-day horde is extremely funny. And since apparently this was automated and allegedly submitted a large number of such patches, it seems pretty plausible this in fact occurred.
I genuinely think that their actions are more likely to represent a divergence of interests with the NSA, which isn't that surprising given recent events. I would be very surprised if they found every zero-day that the NSA already knows, but this probably doesn't make them happy. Anyway, now that Mythos is out of the bag, most (intelligent) devs are going to be giving their code closer scrutiny, regardless of whether they have access or not. Arguably, they should have started last year.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
If you are a programmer I recommend clicking through to the referenced red team blog and reading some of the technical details they have revealed. "Crash any OpenBSD host with carefully crafted TCP packets" seems pretty bad. And finding bugs in cert libraries where they only verify that DNs match rather than verifying thumbprints is a classic.
Reading the OpenBSD bug all I can say is that an implementation in Rust wouldn't have this issue.
More options
Context Copy link
Those are important bugs, and I am glad they've been fixed. They demonstrate an impressive level of capability.
But I was picturing multiple simultaneous Spectres and Heartbleeds, which would have been horrifying. I am grateful that this is more wakeup-call tier.
More options
Context Copy link
More options
Context Copy link
Yeah, anti "AI works for coding" person in the top level 2-3 below this one, how do you explain all this? Note that they are providing cryptographic hashes of claimed vulnerabilities today, so we'll see within the next few weeks what these vulnerabilities actually are and if they're trivial we'll all know. Finding a 27y old vulnerability in FreeBSD is up there next level skillz.
Also @self_made_human, you're a regulated doctor, you're one of the least cooked people out there, you'll be protected by laws and regulations long after the rest of us are on the dole.
I'm not skeptical about every single aspect of AI, my main skepticism is over its ability to build and maintain complex systems (usually in the form of codebases that are more than a basic bitch CRUD app). Finding vulnerabilities is definitely something I've always thought was within the capabilities of AI, my biggest concern is the signal to noise ratio. So I'm curious how many false positives Mythos found that they had to filter through to find the 4 examples they list as ones it actually found.
There's a cost aspect as well. If it costs $200,000 to find a glitch in a video codec that may, horror of horrors, cause your player to crash (and which, to anyone's knowledge, hasn't done so in 16 years), that's not exactly a selling point. $200,000 may actually be an understatement; they said it took 5 million tries to catch it. At 20 cents an attempt, more like a million dollars. We also don't know if they ran any of these tests on old code with known bugs. If they did and the software didn't catch half of the ones that were already caught, its utility isn't that great.
I wish the AI skeptics would limit themselves to forms of naysaying that aren't contradicted by the press release!
That's not what they said. They said five million runs of existing automated testing tools (fuzzers) didn't catch it.
They explicitly mention their hit rate by severity versus opus:
More options
Context Copy link
Operating system and browser zero-days go for millions of dollars.
If Mythos can spit these out for a million dollars a run it's still extremely scary.
This is only true in the darkest of gray markets. In the white-hat arena that Anthropic would be forced to bargain in, these exploits go for 10s of thousands.
The military is of course willing to pay black-market rates, but Anthropic kinda burnt that bridge... and I'd be honestly pretty surprised if In-Q-Tel (famous CIA front company) starts investing in Anthropic...
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Per the quote, it was OpenBSD, which is an operating system with a very strong focus on security. (By reputation, I am not paranoid enough to run their OS personally. I do run their ssh server, like everyone does, and have no complaints except for that one Debian 'fix', and I can't blame Theo et al for that.)
Ok, that's even more impressive than finding a vulnerability in FreeBSD.
OpenBSD still finds a buffer overflow every year or two. It's definitely better than 95% of big software projects out there, but it isn't perfect. Definitely not trying to minimize what Mythos actually found though.
https://www.openbsd.org/errata74.html
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
They are possibly very cooked, because AI is a lot better at day to day doctor tasks than it is at mathematical STEM, and telehealth is already on the rise. Economic disruption as severe as „every tech worker on the dole“ will result in political shake-ups, like universal healthcare in the USA and huge cuts to unpopular physician monopolies, resulting in huge salary declines for doctors.
Best I can tell the LLMs have basically found use as a "force multipliers" for skilled workers to expand their productivity, especially in finance and tech. This news exhibit an extension into searching a solution space with later verification by skilled workers. I'm sure the use cases will continue to expand but medicine is fundamentally different - you'd be looking at replacing a skilled worker for purposes of replacement (obviously) and unlike other cases were someone verifies, in a replacing doctors scenario you'd need to be getting it right 100% of the time with no second check. In medicine the checking would be the same as doing the work.
Which doctor manages that?
Yeah, I know, that sounds like an insult/joke, but estimates of iatrogenic death rates in US hospitals are at minimum 20k/year out of 700k, which means that in even in high-stakes scenarios doctors and nurses are only at 97% for "getting it right enough not to kill someone"; fully "getting it right" would be a much higher bar and lower success rate. All the AI has to do to replace skilled workers is get more reliable than they are.
You're surely right that AI in medicine isn't as good a replacement for human workers as AI in fields where checking results is easier than producing them ... but it's perhaps similar to AI for self-driving cars: stringent requirements and potentially-lethal consequences, but the AI still doesn't have to be perfect to be an improvement, it just has to be better than the typical human competition.
The medication error death rates thing is last I checked pure unadulterated bullshit - literally not statistical analysis or research with severe methodology flaws (ex: multiplying small scale studies from other decades over the entire population decades later, mixing together preventable errors with unavoidable adverse events).
As an oversimplified example - if you were already dying of a stroke and were given a medication that prevents death 90% of the time and affirms death 10% of the time (via for instance a bleed), they'd mark that as a medication error and add it to the killed by medicine pile. That is....stupid.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
"Least cooked" and "very cooked" are significantly overlapping distributions. Pretty much everyone who isn't ready to FIRE or is independently wealthy (and maybe politically connected) is potentially cooked. And that's assuming aligned AGI, or else you better hope you make for a particularly pretty paperclip.
Anyway, as that joke goes, we're all dying, some of us are just dying faster.
More options
Context Copy link
More options
Context Copy link
So I hope, but it's far from granted while I work for the NHS. Rishi Sunak threatened to cut costs and put uppity doctors in their place by augmenting mid-levels with AI a few years back, and was laughed at. Even I don't think the models of the time would have been good enough. But times have changed, while the NHS and its only becomes a more tempting target for financial bariatric surgery (and the models have gotten much better). Starmer probably won't be the one to make the call, given his politics, but desperate times call for desperate measures.
I'm confident it'll happen eventually, and far too soon for comfort. The average man, the kind staring at double digit unemployment figures or laid off themselves, would have pointed questions about why doctors and other regulated professions are let off the hook. I think it only buys me like 2-5 additional years of security at best.
And in India? Haha. Sadder haha. It's going to be a bloodbath and the service sector is not going to have a good time. The economy it props up? You connect the dots.
More options
Context Copy link
More options
Context Copy link
Biggest red flag to me that this is more marketing puffery overselling capabilities than reality:
I.e. "This AI could be utterly devastating even if we only let it loose on our internal network. We'd better be super duper extra careful and cautious before we let it loose. 24 hours ought to be fine, what could we possibly miss in such a massive time window?"
If that's the biggest red flag you can find?
Well, I mentally include you in the list of skeptics too, so I've already wished you luck.
(And I suppose I should thank you for listening to others when they asked you to try repeating your recent experiment with Opus instead of Sonnet. That makes you a better skeptic than many I have the displeasure of knowing on this forum.)
More substantively:
Anthropic takes misalignment seriously, though concerns were raised after the loosened their RSP. You can't really evaluate the safety of the latest and greatest models while being maximally restrictive, at least not if you don't want to be scooped by your competitors with fewer scruples. Anthropic acknowledges this tension explicitly, and asks for forgiveness for moving with haste even they aren't quite comfortable with. I can only assume that reasonable care was taken to minimize the scope for danger even when they did a wider internal rollout.
Plus, they've already said they're not going to make Mythos public, even if some of the benefits will trickle down to the next Opus. That is not something a company that is desperate for money or willing to ignore safety would do.
Oh, Boo. You can bet your ass the military and big corpo will have access to Mythos, why can't the ordinary man get it too (even for the appropriate fee including a fair margin rate on top of their development + running costs).
The same reason why you're not allowed to own a machine gun.
More options
Context Copy link
@ChickenOverlord is correct to point out that Anthropic has only said they won't release Mythos Preview, but that they're planning to release "Mythos-tier" models eventually, when they deem it safe.
More options
Context Copy link
More options
Context Copy link
This is a little bit of a hijack, but it's topical. About six weeks ago, you offered to run some coding tasks on a frontier model. Did that ever go anywhere?
Unfortunately, not yet. My collaborator (who would, let's be honest, be doing the heavy lifting that isn't handled by Opus, and certainly more than me) was disappointed by the general quality of the submissions. Not all of them, of course, but many of them seemed too demanding or out of scope. Both of us have also become far more busy, and I have no intention of chasing him regarding it. I did nudge him a few weeks or two back, and that's what he told me.
This doesn't necessarily mean that it won't happen, but you shouldn't get your hopes up. I can't really do it by myself, I have no reason to pay for Claude Max, leaving aside technical capabilities.
There were a few ones, now that I think about it, that I could handle by myself, but they're not the most impressive examples. And I do genuinely have a lot on my plate.
More options
Context Copy link
More options
Context Copy link
I mean I'm sure I could find others if I tried.
Thanks, I try.
They've only said the preview of Mythos won't be public, the final release will be.
A little ambiguous, but the following makes it sound like a limited release for certain partner companies.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link