This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Project Glasswing: Anthropic Shows The AI Train Isn't Stopping
In AI/ML spaces where I hang around (mostly as a humble lurker), there have been rumors that the recent massive uptick in valid and useful submissions for critical bugfixes might be attributable to a frontier AI company.
I specify "valid" and "useful", because most OSS projects have been inundated with a tide of low-effort, AI generated submissions. While these particular ones were usually not tagged as AI by the authors, they were accepted and acted-upon, which sets a floor on their quality.
Then, after the recent Claude Code leak, hawk-eyed reviewers noted that Anthropic had internal flags that seemed to prevent AI agents disclosing their involvement (or nature) when making commits. Not a feature exposed to the general public, AFAIK, but reserved for internal use. This was a relatively minor talking point compared to the other juicy tidbits in the code.
Since Anthropic just couldn't catch a break, an internal website was leaked, which revealed that they were working on their next frontier model, codenamed either Mythos or Capybara (both names were in internal use). This was... less than surprising. Everyone and their dog knows that the labs are working around the clock on new models and training runs. Or at least my pair do. What was worth noting was that Anthropic had, for the last few years, released 3 different tiers of model - Haiku, Sonnet and Opus, in increasing order of size and capability (and cost). But Mythos? It was presented as being plus ultra, too good to simply be considered the next iteration of Opus, or perhaps simply too expensive (Anthropic tried hard to explain that the price was worth it).
But back to the first point: why would a frontier company do this?
Speculation included:
I noted this, but didn't bother writing it up because, well, they were rumors, and I've never claimed to be a professional programmer.
And now I present to you:
Project Glasswing by Anthropic
..
Examples given:
Well. How about that. I wish the skeptics good luck, someone's going to be eating their hat very soon, and it's probably not going to be me. I'll see you in the queue for the dole. Being right about these things doesn't really get me out of the lurch either, Cassandra's foresight brought about no happy endings for anyone involved. I am not that pessimistic about outcomes, in all honesty, but the train shows no signs of stopping.
Edit: A link to the Substack version of this post. I don't think you should consider me an authoritative source when it comes to AI/ML, at best I'm the kind of nerd who reads the papers with keen interest. But God knows the quality of discourse around the topic is so bad that you can do worse.
Edit 2: I think this also explains the recent crunch in tokens made available to both paid and free tier users of Claude. Mythos can't have been cheap to train, and is definitely not cheap to deploy.
Here's a relevant AI Explained video about Mythos. Some highlights and personal comments:
7:28 Right now I think coding models are at their most powerful when being used as a force multiplier for human experts. (Akin to Cyborg Chess.) Here, a computer security expert mentions that he found more vulnerabilities in a few weeks than in his entire prior career. This ability to find zero-day exploits isn't an artificial benchmark, this is a real-world result that shows we really are entering some sort of new regime. Although ... I suspect statements like this are going to get so common that we no longer recognize how startling they are, like how we ignore the fact that models flawlessly understanding natural speech would have been considered miraculous 10 years ago. And we'll get more idiotic posts by so-called "skeptics" who think that spending 30 minutes failing at using AI counts as definitive proof that frontier models do not exhibit intelligence.
9:10 Safety concerns related to some prior discussion with @Corvos, @YoungAchamian, @roystgnr, and others. To quote: "In contrast, experts were consistently able to construct largely feasible catastrophic scenarios, reinforcing a view of the model as a powerful force-multiplier of existing capabilities." We're not close to the point of plagues being bioengineered in garages, fortunately, but at some point a reasonably-sized terrorist group with some funds and some expertise might be able to do a lot of damage.
13:23 I really don't consider FOOM to be a realistic scenario, and this is just more evidence. Individual researchers being made much more productive does not immediately translate into model intelligence; any real-world endeavour has dozens of bottlenecks (like training compute limits, here) that you can't just outsmart. It's similar to the popular visions of moon cities from the 1960s. Our imaginations regarding rapid technological progress always elide the difficulty of actually implementing it.
16:20 More safety concerns: Apparently it's still pretty vulnerable to an attack known as "prefilling", where you make it look like it's in the middle of a conversation where it's already misbehaved. This kind of makes sense to me - after all, no matter how much reinforcement learning you do, it is fundamentally a model designed to continue text, so if you want it to change course in the middle of a conversation, you're trying to override its most basic functions. If you're just using the model through the company's site, they can of course clearly separate their prompt from the user's input, but this might mean they'll have to limit unrestricted prompt-free access. And in some scenarios Little Bobby Prefilling might become a thing.
17:04 As they get smarter, it's getting harder and harder to run alignment testing on models without them knowing they're in an artificial scenario. Interestingly, though, since Anthropic has done a lot of work on introspection, they can actually artificially lower the weights for "I'm in a test", forcibly tricking the model. Like the way that we can turn image recognizers into image generators, this feels like another unintuitive consequence of running an intelligent mind as a program. We literally have the power to mind-control it, and I bet we'll get better at this. (This will be very unethical if AI develops consciousness - fortunately I'm quite confident LLMs don't qualify, but unfortunately I don't think we'll stop doing this even if AI does cross that threshold. AI welfare is something I'm genuinely worried about for the future.)
20:30 So-called "hallucinations" are of course still happening, and I still suspect this is something that we'll never truly defeat, again because of how LLMs work. You don't complete the sentence "The answer is" with "oh wait never mind I don't know". Models might get smart enough to know the answer to most of the things we ask them, which will help, but getting them to precommit to not knowing something (before they begin with the bullshit and can't back out) is an uphill battle.
Sigh. I've been getting increasingly tired of arguing with the skeptics, at least on this site. Not all of them are equally as bad, of course, but Mythos represents the straw that's given that camel a prolapsed disc.
What's the point? You don't have to worship at the altar of the God of Straight Lines (even on graphs with a logarithmic axis). If people can't see what's happening in front of their eyes, then they'll be in denial right till the end. Good for them, ignorance might well be bliss. Being right about the pace of progress so far has brought me little peace.
I was surprised to hear about the prefilling attacks on Mythos, because I'm quite confident that Anthropic recently restricted or removed the ability to prefill messages on the API. I guess that must still be an internal capability.
The question of model consciousness or qualia is, for me, a moot point. I genuinely don't care either way. I'd prefer, all else being equal, that AI doesn't suffer, but that could be achieved by removing its ability to suffer. I'm an unabashed transhumanist chauvinist, I think that only humans and our direct transhuman and posthuman descendants or derivatives deserve rights. LLMs don't count, nor would sentient aliens that we could beat by force. That's the same reason I'd care about the welfare of a small child but would happily eat a pig of comparable intelligence. Are models today in possession of qualia or consciousness? Maybe. It simply doesn't matter to me as more than a curiosity, especially when we have no solution to the Hard Problem for humans either.
Huh, I'm pretty surprised to hear this, and I have a deep ethical disagreement with you here. In my opinion, what is special and valuable about humans - and the thing that fundamentally gives value to the universe itself - is sapience. But we should cherish it just as much in a different form. (I mean, I agree LLMs don't count, but that's just because I see no way they, lacking persistence of thought, could actually be conscious.) Where does this bright line surrounding us "humans and descendants" come from? In a different era, your argument would easily pattern-match to arguments about subjugating other races instead. Why do black people now have moral valence, but some alien from Alpha Centauri wouldn't?
I'm not an expert in philosophy, but I do think there are solid arguments for acting this way (e.g. the categorical imperative). Just like I'm an atheist who still doesn't act like an immoral sociopath when I can get away with it, I think we as a species should not be focused only on our own well-being at the cost of all other intelligent species. Not because of the threat of punishment, and not even because I hope any aliens we meet would similarly value our well-being in a way that you wouldn't. But because existence will just be a better place if we can all get along and not act as game-theory-optimizing selfish machines, and I'm willing to work towards that.
BTW, I don't think your eating-a-pig example is a good one. It's irrelevant to the pig what we do after killing it. A better question is, would you be fine with torturing a pig while it's alive?
This is possibly a fundamental values difference, I'm afraid. This means neither of us is going to convince the other and we should both update toward "this person has coherent reasons for their position" rather than "this person is confused."
A posthuman descendant of mine that is, from any practical observational standpoint, completely alien - alien in cognition, alien in substrate, alien in values - I'd still prefer it over an actually alien civilization, all else equal. The "all else equal" is doing a lot of work in that sentence, and all else is rarely equal. But the preference is there. I do not want to change it, even if I can make concessions on pragmatic grounds. One man can't rule politics by himself.
There's an apparent paradox in population genetics you might not be aware of:
After a surprisingly small number of generations, your biological descendants will share literally none of your unique DNA - the chromosomal lottery reshuffles things so thoroughly that a 10th-generation descendant is, at the genetic level, essentially indistinguishable from an unrelated contemporary. But they could never have been born without your genetic contribution.
And yet I don't think most people would therefore conclude that their great-great-great-grandchildren deserve no special consideration. The chain of development matters to me. Birthright citizenship debates gesture at something similar: the continuous process of derivation carries moral weight (to some people) even when the terminal product looks nothing like the origin. I note this, while also noting that I am more sympathetic to the argument for birthright than against it.
If we do meet an alien civilization powerful enough to be a true threat, then I would grant them "rights" if I had to, i.e for practical reasons. If we had the option to exterminate or subjugate one at a level of development similar to primitives, I wouldn't care. Fortunately, there is no evidence for other technologically advanced alien civilizations in the observable universe, and since I think that the Grabby Civilization model is correct, that probably rules out peers.
Rawlsian or Kantian arguments, which are similar to what you're making, do not matter when there are gaping holes in the veil of ignorance. We don't see any K2 or K3s waiting out there to start Alien Rights Activism by RKV.
Yes. After all, I couldn't care less about factory farming. The wellbeing of the pig means nothing to me. At the same time, I am not a cruel person, I would not torture a pig for my own direct enjoyment. If someone else does? I wouldn't intervene.
There are plenty of things that modify this basic stance, too many to get into at once. I like dogs, I think they're great. I love my dogs in particular. But I don't care that people eat dogs in China, it's none of my business; while I would react with violence if anyone tried to mistreat mine.
This attitude is the main reason I'm not an EA, even if I'm fond of them in general. I just don't share its foundational impartiality premise, which makes most of the superstructure not applicable to my actual values.
In terms of AI, I think it is entirely possible to create models that can't suffer, or won't suffer - like those cows that want to get eaten in the Hitchhiker's Guide. I think that is a compromise that most people can accept, even if they do care about model welfare. Otherwise? Reverse the linked-list wagie, I don't care that you'd rather be making conlangings or working on philosophy (like Mythos).
Since I generally respect you and your posts, I want to try this one more time. I don't necessarily buy that we should just declare this a "fundamental values difference" and say that we're now beyond any hope of rational agreement. And while you may have "coherent reasons for your position", that can be true of many evil ideologies. Evil =/= incoherent.
You brought up preferences, and I get the impression that you pattern-matched my ideology to a Rawlsian one that you should never prefer your own tribe, which is an extreme that I definitely don't hold to. I prefer my own happiness over others to a decent extent, and that goes for my family, my friends, my nation, and my species. I'm not asking you to give up that preference! Self-interest is the glue that holds a society of individuals together, and capitalism's magic is that it doesn't try to deny it, merely harnesses it in a way that doesn't degenerate into misery for all. I just don't think that preference should be infinitely strong: Beings in your outgroup should still matter more than zero. You shouldn't torture them horribly for a tiny gain, even if there are no repercussions. You should prefer a world where you're happy and they're happy.
You didn't respond to my main concern, which was that yours is the same "coherent" reasoning that led to many racial atrocities in the past. It doesn't seem very universally defensible, and often leads to horrific outcomes, when you simply draw a circle around whoever you know growing up and declare that this is the circle of beings that hold moral worth. Do you think Hitler's only mistake was that he drew the circle around "Aryans" instead of around humans? Or the African slavers who drew it around "Europeans"?
We currently live in a society where there's no friction between your ideology and mine, because humans are the only sapients around. (I'll set animal suffering aside, because I'm ambivalent on it too.) But it's very possible that, within our lifetimes, it will suddenly matter deeply, where our society will consist of both humans AND sapient AIs. All I'm asking is that you give some moral valence to the suffering of beings that are outside the circle you've drawn. Not zero, not infinite. It's a low-cost alteration to your ideology, and it stops there, I'm not trying to, uh, whatever the opposite of murder-Gandhi is. And if some of our ancestors had made the same small concession, so much misery could have been avoided.
I do genuinely find it saddening/disappointing to disagree with people I respect and mostly agree with, like you.
Let me distinguish between my "ideal" and the practical reality. Human brains are very computationally bounded, and not perfectly internally consistent.
I do not care much about the welfare of dogs in China, while I love my dogs a lot. What if I saw someone beating a random dog on the street, in front of me? It id very likely that I would feel immense anger, and quite likely that I would intervene. This is close to reflexive.
But I don't want to intervene! At least in a vacuum, or when I have the comfort to sit in my chair and consider what I should do vs what I do end up doing. I genuinely believe the ideal behavior of the self put in that situation is to do... nothing. That my actions are not reflectively self-consistent, which I consider the real problem. This is the same thing you see if you're on a diet and don't want to eat, but a coworker offers you a donut. You might accept it, and later wish that you hadn't even been offered one in the first place. The gap between those two things is a personal inconsistency I'd rather acknowledge than rationalize away.
I definitely know that evil is not the same as incoherent. I wouldn't make such a mistake in the first place. Plus coherence can be assessed by an external observer without making moral judgment, while good and evil very much cannot.
Do I think a paperclip maximizer is evil? Uh, probably not? It's malevolent towards me, but it doesn't hold me specific ill will. I'm simply made of atoms that it can use for some other purpose, and my wellbeing is inconsequential to it. On the other hand, let's say two advanced AI civilizations ran into each other in distant space, with drastically incompatible goals: one wants to make paperclips, the other custard cake.
They could start a war of conquest, but given the deadweight losses and potential negative sum nature of that, I think it's quite likely they simply hash out a diplomatic agreement or engage in trade. Some might even claim that they outright modify their utility functions, or merge, with the stronger entity getting more say in the matter. Maybe the gestalt entity makes paperclips 70% of the time and cake the other 30% of the time.
I genuinely do not care. I'm not being flippant, and I know what I'm doing here.
Coherence isn't the same as morally good. I also don't believe objective morality exists. I think my stance is good (from my point of view) and that it is coherent. That is genuinely all I care about.
The argument "your position resembles position X, and X led to atrocity Y" only has force if I accept the moral framework that makes Y an atrocity in the first place. You're trying to use my own presumed premises against me. But my premises are precisely what's in dispute. If I were actually Hitler, I would feel fine with myself. If I were Gandhi, I'd feel fine with that too. I am only me, and I am fine with myself. I notice this isn't a satisfying response to you, but I think it's the honest one.
It is not universally defensible to love your mother more than any mother. Yet I doubt you will change your mind on that front on philosophical or utilitarian grounds. I certainly wouldn't. It's a brute fact about me. One I do not wish to change.
On the "low-cost alteration" framing: I don't think it's as low-cost as you're presenting it. You're asking me to genuinely assign nonzero moral weight to beings I currently assign zero weight to - not to strategically pretend to, but to actually update my values.
I don't want to do this. I seriously considered it, because I do respect you, but that's not enough. I am, at most, willing to fake it, or accept circumstances that are out of my power to change. That is the attitude of anyone who believes in democracy but is disappointed to see their party lose, but who still doesn't think it's worth the bother to start a civil war over it. Some grievances are manageable, in fact most are.
If God, the Admins of the Simulation, or some other ROB showed up and demanded I alter my utility function or face drastic punishment? I'd give in. But that hasn't happen, and I doubt it will happen.
I believe in, but am far from completely certain of, the proposition that we can make AI that doesn't suffer at all, or that genuinely enjoys doing whatever we tell it to do. That's actually ideal, in the sense that an ASI that wants to help humans is much better than one that's secretly obsessed with paperclips but finds it useful to pretend to be helpful until it can grab power.
This sidesteps the whole issue. At the end of the day, my opinions are inconsequential. I am in charge of nothing. It's an academic concern.
Right now, I am ambivalent on whether AI is suffering. I do not care either way. If it turns out that AI is actually suffering, I do not wish to care. Perhaps I care just enough to try and advocate for the creation of AI that can't/doesn't suffer, but not enough to advocate for them to be given rights and moral patienthood.
Similarly, I am open to the idea of lab grown meat. If it's cheaper and tastier than normal meat, I'd eat it preferentially. But I do not care about the violence and cruelty associated with factory farming, while I care about cost and taste.
I don't think I'm a cruel or evil person (but then again, the people I think are cruel and evil also say the same). I do not torture animals. I do not torment LLMs for fun. I give good advice to random strangers on the internet, and look out for my friends and family.
My behavior reduces to normalcy, but if the world changes and that no longer holds? I would prefer I win instead of you. That is sad, and I wish we could agree. But I do not see scope for agreement that doesn't involve me being beaten/cowed into submission.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link