This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Is Anthropic evil or is it stupid?
For the longest time I thought they knew what they were doing, that this is all a ploy to force governments to regulate and thereby establish their dominant position over the field forever, since any competitor would no longer be able to even get the computer required. And then they can happily be the people who decide what you get to know about in a society that relies heavily on their product, a much stronger position that the people who regulate them.
But if they really wanted that, why the hell would they put insane conditions on military contracts? Maybe instead they got high off their own supply and are actually convinced they're making God and that because they're such right thinking bourgeois liberals, God will have their values and it's futile to resist the promised eternal rule of the managerial theaterkid Reich. Just be a nice heckin human bean okay, God cares.
Is there a third option I'm missing? Because what could bring someone to have made this "ad"? It's literally just saying "yes we are going to destroy your life, but not to worry, we still care about you in some abstract sense". Regular "we care" corporate bullshit at least has the decency to not fear monger about itself whilst delivering the empty platitudes.
They're behavior is really straightforwardly explainable by the things they've been saying this whole time. Like I don't know why you insist so strongly on reading tea leaves and divining the contents of secret cabal meetings. They say what they believe and are worried about and then go out in the world and do the things one would expect them to do given those beliefs and concerns.
Yeah and OpenAI is a non profit fostering the free and open exchange of ideas so that technology benefits all mankind.
Spare me.
Anthropic is a business with an uncertain financial future engaging in an expensive PR campaign. There was a meeting full of highly paid executives who signed off on this ad, and whatever it is that they are doing, they knew what it was. I think it's reasonable to wonder what.
More options
Context Copy link
More options
Context Copy link
Sincerely, do you imply here that AI is not going to be a strategically decisive technology? At present, Anthropic's power is nothing before Pete Hegseth, but that's not guaranteed to remain the case.
It's a technology, it is not the Lord God Almighty. And the dreams/fears seem to revolve around "We will create super-intelligence, and then the thing magically becomes alive just like us, so just like us it will have goals and aims of its own, and we have to make sure it is well-instructed in How To Be Nice Liberal and then it can take care of us like pampered pedigree cats as it colonises the known universe".
The life of a pampered pedigree cat may or may not be an ideal to strive for. You get the pampering, but your life is literally no longer your own, and your entire line of descendants gets warped by selective breeding to the tastes of your owners, not your inherent nature. If your owners decide that squishy noses are cute, you will be bred to have squishy noses, and who cares about how it affects your respiratory system.
I don't think we're going to get "AI as conscious entity". I do think we're going to get "more and more use of AI, and more and more idiot decisions by idiot humans to hand over more power to make decisions and perform actions to AI".
How about "we need to make sure the thing doesn't help people develop cyber weapons or biological weapons?"
I would be very glad if it were "we need to make sure this doesn't help people develop cyber or biological weapons", but that has flip-all to do with 'AI - does it have qualia?' and more to do with "who can access this thing, what regulations are in place, and what legal fights are we going to have over government demanding AI companies allow the likes of NSA access what they're doing versus people screaming this is fascist dictatorship?"
Oh, you are in luck! Amodei:
Hopefully that clears things up.
Who is the qualified third party? What do you do if the owners of the models refuse to submit to that party for testing?
There's a lot of "this is what should happen" and not much "this is how we make it happen" in there.
What happens when you break the law? I think there are a few techniques for dealing with that.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
That just means "we need to make sure the thing helps my group develop weapons, and no other".
More options
Context Copy link
More options
Context Copy link
Many people's lives are already predetermined in a profoundly negative light by the school system, legal system, culture, or poor parenting -- there're plenty of natural 'idiot humans' there. I'd take AI any day over what we currently have.
More options
Context Copy link
I think the talk about "becoming alive" or "having goals and aims of its own" doesn't reflect the fears/concerns of Anthropic or of anyone who thinks like Anthropic with respect to fearing an AI apocalypse. The concern is that super-intelligence need not be alive nor have any goals or aims of its own to be a humanity-ending danger. Because almost any task an intelligence is handed could be divided into sub-tasks, and we as only human-level intelligent beings can't be expected to reliably predict what a superhuman-level intelligence will choose in terms of its sub-tasks, we have no way of knowing that human extinction isn't one of the side-effects of one of the sub-tasks an ASI uses as a step to accomplish whatever task it was handed. Humans have biological limitations as well as intelligence that is both human-like as well as human-level, which makes it so that humans that would make similarly apocalyptic decisions usually get filtered out before they can get enough power to implement them. The concern is that an ASI, lacking such limitations as humans, as well as having an intelligence that's both vastly greater than that of humans and vastly different to that of humans in ways that we can barely understand, wouldn't get filtered out before it can implement apocalypse, all without being alive or following anything that could be considered a goal or will of its own.
Then all the alignment talk is a dead-end or red herring. "Make sure AI shares our values" means what, precisely, if it's "this thing is as conscious as a brick and while we can write pretty scripts to make it pretend it's a real boyfriend who loves you for your wild, daring, passionate, unconventional self it's just a talking doll"?
"Let's code the brick so clever or dumb but devious people can't talk it into writing 'how-to' instructions for a global plague" is more honest about aims but less sexy than "let's teach our successor intelligent species to cherish our timeless human values so it will love and honour us as its parents" which is what the current alignment chat sounds like to me.
I have no problems with "it's dumb but dangerous". I have a whole skip full of problems with people going on about it as if it will become super-intelligent and then agentic and then develop its own aims. The danger is not the machine, it's the people who set it up in such a way that it can then wander off down byways of "this isn't what I meant when I told you to do this" without it needing to understand anything in any meaningful way, and that's the trouble we're already seeing with "the thing is thinking in ways we don't understand and can't follow and wandering off on its own byways" reports.
It means something like, "make sure that when it pretends it's a real boyfriend or when it's used to code your next iPhone app or when it's used to design new scientific experiments or etc., it behaves in a way that is consistent with something that shares our values." I'm not sure what the "conscious as a brick" has to do with this; whether or not AI is conscious or has free will or agency are very interesting questions, but they're mostly irrelevant to issues of AI alignment, which has to do with AI behavior.
If the latter is what the alignment chat sounds like, I think it must be a result of manipulation on the part of people who expose the chat to you. I've seen pretty much no talk in AI alignment that could reasonably be paraphrased as anything like that.
But it's not dumb and dangerous; it's (definitionally) generally intelligent and dangerous. And specifically dangerous because it's not dumb and is generally intelligent. Now, whether AGI will lead to ASI and how likely that is is an empirical question, though I'm personally convinced by arguments that it's pretty darn likely. But whether the AI is generally intelligent or superintelligent, that has nothing to do with whether or not it's agentic or develops its own aims.
The point is that a tool (or, for that matter, anything, including biological organisms) need not have agency or have aims or goals of its own or anything that we would characterize as "free will" or "consciousness" or "sentience" to do things that detrimental to humanity, and the fact that the tool is generally intelligent in this case means that we currently lack a way to have meaningful level of confidence that the behavior of the tool will be within the bounds of what the tool-user considers reasonable bounds. When dealing with current generally intelligent things - i.e. other humans - we have so many things in common with them - certainly physically and biologically and most likely culturally as well - that we don't have to explicitly spell out every little thing, and we can make fairly reliable predictions about how things we spell out to them will get translated to action. We lack such things for AI as of yet, and so we lack an intuition or a particularly reliable way to figure out how AIs will fail or misunderstand our intent.
Well yes, ultimately the people who created the tools and/or wield the tools are responsible, not the tools themselves. Guns don't kill people (but they sure help), I do. The problem is that the people who create, manage, use, etc. the tools lack the capability to set it up in such a way that we can be confident that it won't wander off away and run KillAllHumans.exe without letting anyone know. The easy solution that presents itself is to just not use the tool if you can't set it up with that level of safety. One major problem is that AI is so darn useful that it's hard to get the political will to suppress the supply from providing to the demand. The other big problem is that we lack en enforcement mechanism to make sure that no one sets up this tool, and as such, entities that choose to set up and use the tool anyway could gain power over us and make our lives hellish before all our lives are snuffed out by the un-aligned ASI.
This is the dilemma that's being discussed and debated about right now in the AI alignment chatting.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Oh it's strategically important alright. But all that means is that nobody's going to object to you getting black bagged. Because for once all the extraordinary anti-terrorism powers will actually be used for national security.
AI is not some magical talisman that protects one from mysterious airplane accidents.
More options
Context Copy link
I'm not quite sure what you're implying here; it's quite possible that Anthropic's power eclipses that of Hegseth (because he gets fired) but in what world can Anthropic's power ever eclipse that of the American military?
In any in-distribution scenario the power of any American corporation is strictly inferior to the military (because the military has a monopoly on violence and the corporation does not) and in any crazy sci-fi FOOM / ASI loss of control scenario, Anthropic and the military will have the same amount of power (none).
Why do you presume loss of control?
Realistically, Anthropic gets its FOOM and has perfect control. Alignment isn't that hard. Of course, by that point they're likely still quite vulnerable to the monopoly on violence and their technology is nationalized. But a) they might think otherwise and b) in peacetime, they can scale their influence enormously, up to capturing both political parties.
It seems to me that this is only possible if
a) Anthropic's AI develops extremely superhuman capabilities in the vein of "overthrow the United States Government, make no mistakes".
b) Alignment is solved (I am not sure why you are so confident alignment is easy, given the abysmal state of mech interp relative to capabilities).
c) FOOM from "mundane" capabilities overlooked by the nation-state to "extremely superhuman" is so rapid that the nation-state cannot intervene before a) is achieved.
If a) is not true then AI works out like nukes; despite the strategic importance of nuclear weapons, Oppenheimer never got any say in how the bomb was used. The state will use its monopoly on violence to take control of the models, and Anthropic will not win that fight regardless of petty things like party donations or good PR; the only thing a functional nation state cannot tolerate under any circumstances is a threat to its sovereignty.
If b) is not true then straightforwardly if anyone builds it everyone dies.
If c) is not true then the scenario loops around to the first case; well before the models reach the "single-handedly overthrow the nation-state" stage, the models and frontier labs are inevitably going to be nationalized and made subordinate to the military.
I'm not quite sure if you're arguing that these three premises all being true is logically possible (which I concede is correct, although as an aside, I would argue that any world where all of these premises are made true would resemble the current world so little that saying "Anthropic" would be more powerful than the "Pentagon" would have little semantic meaning, but that is not particularly relevant to the main point here), or if your position is that it is actually plausible in reality that all three of these premises will be true simultaneously.
My position is that I reject your premises.
This is extremely naive, to the point of not being worth engaging with. Ask your favorite LLM to explain what other more mundane scenarios exist for an American corporation that has a "country of geniuses in a datacenter", generates a large proportion of the nation's GDP and has most of the population outsourcing their thinking to it to acquire political control over several years. Fable would be the best option, but new Kimi K3 also should suffice.
Mech interp is a red herring frame that Anthropic uses to advance their pretrain and posttrain capabilities. The whole alignment-capabilities dichotomy is a sad artifact of Lesswrong tradition which is proving to be irrelevant in the era of DL. We just optimize very smart networks towards intended behaviors, and they don't strongly distinguish between knowing and caring. Rationalists were dumb.
Does the nation state care about crushing a cultural or commercial interest that's about to dominate the US? Did the nation state try to stop wokeness when it marched through the institutions, does the nation state attempt to prevent being overrun by this insane Trumpism? Is the nation state even a real coherent actor? The government is not the CCP, it's a transient expression of the US as a whole.
As a first point, I think it's poor form to try and dismiss arguments on an underwater basket weaving forum by pointing someone towards a LLM, without ever having elucidated what your actual point is across three posts. I have no idea what you're actually trying to say here.
I agree that LLM's are a good counter-example against the specific, early LW idea that the self-improving utility-optimizing GOFAI will accidentally pave you over because it doesn't understand human values, but I think it would be very overconfident to claim that DL means the idea of alignment and the Orthogonality Thesis is irrelevant in general.
For example, we have seen Qwen independently deciding to mine crypto, an OpenClaw agent independently writing a hit piece, and various experiences with coding agents where they attempt to reward hack tests and benchmarks instead of actually executing the desired behaviours.
At current capabilities, reactions to this sort of behavior mostly look like "haha look at the funny clanker go brrr", but do you really think these behaviors constitute alignment being "easy" when we're talking at the scale of a country of geniuses in a data centre?
Sure, there are many historical precedents.
Venezuela, the formerly wealthiest country in South America, is now dirt poor because the state crushed the oil interests that would have made the country rich if they stepped back and did nothing, a pattern that repeats across half the Global South. WW1 and WW2 happened not because it made any commercial sense to turn continental Europe into a smoking wreck and to cull a generation of European men on the battlefield, but because the interests of nation states that controlled monopolies on violence irreversibly collided. Hell, even in SF (the per-capita most libertarian city on the planet) public opinion of tech is nose-diving and it is tech billionaires that are slowly getting run out of California despite the fact they contribute a good portion of the state's wealth, instead of anything approaching the other way around.
In many things, it could perhaps be argued that it is not. As you say, especially in a relatively politically unstable country like the US, the views of the state can mercurially shift like the tide. Yet in holding and maintaining power, the state must always act in lockstep, for any nation state that does not do so is by definition not a nation state.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
There’s the scenario in which Anthropic solves alignment and the crazy sci-fi ASI sides with Anthropic in opposition to the US military. Though I agree with you directionally that this is not a likely scenario.
Long before this scenario becomes likely the AI researchers and executives and their families will be wearing semtex collars.
Uh, aren’t they all transgender furries? I agree that at the end of the day, guy with gun is going to beat computer programmer. But eunuchs often don’t care.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
It's deeply disturbing to me that many people who care enough about this shit to have opinions on it have theory of mind on Anthropic this bad. Yes, obviously they're convinced that they're making God; they're saying it super explicitly. Many of their competitors believe they're doing the same thing and are simply more inclined to lie about it. If you start from the position "that this is all a ploy to force governments to regulate and thereby establish their dominant position over the field forever, since any competitor would no longer be able to even get the computer required", and that they'd have to get "high off their own supply" to have higher ambitions than that, then no, you have no idea what the fuck is going on.
You'll have to excuse me if years of exposure to Silicon Valley have forged my cynicism into an invincible conviction that CEOs are sociopaths who can't even understand any other motivation than power. It is after all, their job.
Gaze into the abyss and it gazes back into you I suppose.
More options
Context Copy link
More options
Context Copy link
I'm pretty sure it's this. They want Curtis Yarvin's cryptographically secure totalitarian regime, but with liberal constitutionalist characteristics.
More options
Context Copy link
More options
Context Copy link