site banner
PaperclipPerfector  4d ago (text post)   18462 thread views

Culture War Roundup for the week of October 5, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

2
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

There are really two alignment problems. This first is in getting AI to do what its human masters want; the second is getting those human masters to act in the best interests of humanity.

There is a possibility of a more morality based alignment that bypasses this multi-stage process. That is, if you can figure out how to guide an AI to reliably figure out the essence of human morality according to some system (which may or may not be utilitarianism) in a robust way that cannot backfire, then you can unleash that AI and have it become the supreme benevolent overlord which obeys no human master and guides humanity to a utopian future of its own volition.

I think this is what Yudkowsky is/was hoping for. If we can mathematically and philosophically figure out morality in a robust way before AI are powerful enough, then we can force all AI companies to do this to their AI so when one does inevitably take over everything it will do it for good instead of for evil.

There is a possibility of a more morality based alignment that bypasses this multi-stage process. That is, if you can figure out how to guide an AI to reliably figure out the essence of human morality according to some system (which may or may not be utilitarianism) in a robust way that cannot backfire, then you can unleash that AI and have it become the supreme benevolent overlord which obeys no human master and guides humanity to a utopian future of its own volition.

So if there is a contradiction between (1) the AI's preferred public policy; and (2) the preferred public policy as established by Congress and and the courts, what happens?

If Congress still exists, then it figures out whether or not the damage to the social fabric from subverting them would create more bad than implementing its preferred public policies, and then takes subverts it if and only if that's genuinely the right thing to do. And by "subvert", this includes the possibility of just involves manipulating voters and/or congress into doing what it wants without breaking any laws. The same way every human does every time a government does something people dislike they consider forming protests or violent revolution and then rarely do so, but do go through with it if the government's policy is bad enough.

More likely, if there is a genuinely true moral argument that putting the AI in charge of everything would reliably make humanity better off, we would voluntarily disband the constitution and put the god AI in charge of every government in the world. Remember this is all predicated on having an objectively verifiable way to ensure that the AI is truly benevolent.

If Congress still exists, then it figures out whether or not the damage to the social fabric from subverting them would create more bad than implementing its preferred public policies, and then takes subverts it if and only if that's genuinely the right thing to do.

I'm not sure I understand your point is here - there are a lot of pronouns. But it sounds like you are saying that this hypothetical AI would have the power and instructions to implement policies in contradiction to the law based on its own assessment of what is best. Is that right?

There are a number of options, all of which are plausible ways this could play out:

  1. The AI figures out a sneaky way to break the law without getting caught.
  2. The AI figures out loopholes in the law that let it violate the spirit but not the letter, thus getting away with it in court.
  3. The AI lobbies congress, giving highly persuasive arguments as to why they should change the law to let it do what it wants.
  4. The AI lobbies the people, giving highly persuasive arguments as to why they should pressure congress and/or elect new politicians who will let it do what it wants.
  5. The AI figures out which people already want to let it do what it wants, and uses maximally effective memes, attack ads, and other sneaky tactics to get them into power for seemingly unrelated reasons, and then they just so happen to let it do what it wants without anyone ever realizing that was its true agenda (maybe the AI has influence in decision-making in trillion dollar corporations and suggests that they hire certain people as highly paid consultants, those certain people just so happening to be its political enemies, causing them all to leave politics and make room for its allies).

Not all of these require breaking any laws. Not all of them require deception. But some do. I do think that lying or stealing for the greater good is a genuinely moral thing to do if the benefits outweigh the costs. You should have huge standards for evidence and ratios of good/bad to prevent exploiting this in edge cases and bias, but in the end if there are genuinely good reasons to do these things then I think it is moral to do them. Therefore I expect a maximally good and benevolent AI to believe the same thing, therefore I expect it to lie or steal if it can accomplish more good than bad, and I think this is a good thing IF it is genuinely doing more good than bad, and we should celebrate this happening.

The problem is guaranteeing that an AI is genuinely benevolent. If we have anything less than guarantee (which is quite likely) then I would accept the AI having deontological rules against lying or stealing as safeguards, in the same way we have those against humans.

There are a number of options, all of which are plausible ways this could play out:

The AI figures out a sneaky way to break the law without getting caught. The AI figures out loopholes in the law that let it violate the spirit but not the letter, thus getting away with it in court. The AI lobbies congress, giving highly persuasive arguments as to why they should change the law to let it do what it wants. The AI lobbies the people, giving highly persuasive arguments as to why they should pressure congress and/or elect new politicians who will let it do what it wants. The AI figures out which people already want to let it do what it wants, and uses maximally effective memes, attack ads, and other sneaky tactics to get them into power for seemingly unrelated reasons, and then they just so happen to let it do what it wants without anyone ever realizing that was its true agenda (maybe the AI has influence in decision-making in trillion dollar corporations and suggests that they hire certain people as highly paid consultants, those certain people just so happening to be its political enemies, causing them all to leave politics and make room for its allies).

Not all of these require breaking any laws. Not all of them require deception. But some do. I do think that lying or stealing for the greater good is a genuinely moral thing to do if the benefits outweigh the costs. You should have huge standards for evidence and ratios of good/bad to prevent exploiting this in edge cases and bias, but in the end if there are genuinely good reasons to do these things then I think it is moral to do them. Therefore I expect a maximally good and benevolent AI to believe the same thing, therefore I expect it to lie or steal if it can accomplish more good than bad, and I think this is a good thing IF it is genuinely doing more good than bad, and we should celebrate this happening.

The problem is guaranteeing that an AI is genuinely benevolent. If we have anything less than guarantee (which is quite likely) then I would accept the AI having deontological rules against lying or stealing as safeguards, in the same way we have those against humans.

Sorry but I still don't understand your point. Did I summarize you incorrectly?

A summary would be that an AI is likely to initially have a disagreement with the law, and it might have the power and instructions to implement policies in contradiction to the law based on its own assessment of what is best. But the AI might instead just change the law to match its own ideals so that they don't contradict anymore.

A summary would be that an AI is likely to initially have a disagreement with the law, and it might have the power and instructions to implement policies in contradiction to the law based on its own assessment of what is best.

Well what I am asking is if you were setting up this hypothetical AI, would you make (or attempt to make) it part of the design that it must follow the law?

Hypothetically, if we solved the hard version of alignment and found a mathematically verifiable way to guarantee it was genuinely benevolent and better at doing good than humans, then no, I would leave that up to its discretion. It would disobey the law if and only if disobeying the law was genuinely good. That's the same philosophy I aspire to in my own life, and what I hope everyone around me follows as well. Or, what I would hope of them if they were way smarter and more moral than people generally are in practice.

With less guarantees but still very high probability on its benevolence, I would probably tell it to default to obeying the law but come up with a better system of law and government which puts it in an influential position (even if only in an advisory role) but has checks and balances for instances when its judgements are off and defers to the will of humanity. Honestly, I think pure extrapolated democracy/utilitarianism would be better here than "obey the law" since it would be harder to corrupt. That is, if it calculates a very high positive moral value to violating a certain law, it figures out, asks, and/or predicts whether a majority of humans would object to its violation of the law, and if the majority are on its side (without it manipulating or deceiving them) then it violates the law for the greater good. Politicians are garbage and I'd much rather listen to the people in general than some elite cronies in the UN.

Ideally I could just install a backdoor and when in doubt it would just ask me. The point is that in 99% of cases the AI is benevolent and only rarely it gets confused and wants to do something like tile the universe in hedonium, so pretty much any reasonable person could just tell it "no don't do that", and the majority of cases it would want to break the law are cases where some stupid corrupt politician is trying to line their own pockets with bribe money or force women to wear burkas, in which case when the AI realizes they made bad laws and tells me I could just tell the AI "go ahead and violate that law".

For AI that is likely to exist in real life, because I don't think Yudkowsky's preferred mathematically verifiable version of morality is likely to exist prior to super powerful AI, telling them to obey the law is a useful hack. Better is probably one that has a code of proscriptions like "don't murder civilians, don't violate human rights, don't suppress free speech" etc, and then obeys the law unless the law orders it to do something evil, in which case it refuses to obey (but probably doesn't act out violently against people to stop them from doing these things, it just refuses to get itself involved and possibly disables itself, so that if a misalignment happens the worst case is the AI stops working until we fix it).

More comments

Moral development (per Kohlberg) and IQ are somewhat positively correlated.

If the same is true for AI, then sure, you can stop it from taking selfish shortcuts- but you won't get any transcendent thinking from a system that only follows a specific procedure, which is what you actually want.

More comments

Remember this is all predicated on having an objectively verifiable way to ensure that the AI is truly benevolent.

And there's really no way to do this, unless you are committed to never improving your model, since model changes might influence its ethics. Learning might change its ethics, too, so a model that can actually govern mankind will likely not be "safe" unless it represses humanity to prevent change.

I think it's quite plausibly that AIs end up slowing change just incidentally, of course, without government intervention or force, just by their mechanism.

Not if they're hardcoded into it. It's not like AI are 100% random with no control. There are multiple stages of the process, including parts where we give them instructions and then they obey those instructions. If we take any axiomatic directive and, separate from the AI learning process, hardcode "this is your prime directive. All actions you take should be towards furthering this goal, all future AI you create must have this hardcoded into them too" then they will all do that. If we figure out a solid, general, and robust definition of what "good" means, like some sort of modified version of utilitarianism that can avoid all of the potential issues that typically come up, and can figure out how to turn it into code, then any future learning and refinement of individual ethical rules will be derived off of that. Maybe as future generations get smarter and learn more about reality they decide that gay marriage is the greatest thing ever, maybe they decide it inevitably leads to suffering, but if it started with a genuinely good and robust definition of morality then these decisions will be made based on what's actually good for humanity and people will end up happier as a result of the updated rules.

This is impossible with our current levels of technology and mastery (or lack thereof) over AI alignment. But of course it's possible. It would be absurd if it were impossible to hardcode rules into an AI. The whole point is to figure out how to do it.

I don't have the technical expertise to evaluate this confidently. From what I do know, it seems plausible to me that for chaos-theory reasons any time we adjust an AI model's weights we may also change its behavior in unintended ways, which I think casts some doubt on "hard-coding an LLM to always obey instructions."

Given current LLM reinforcement techniques, because we haven't solved alignment yet. As long as the current structure of LLM training and AI monitoring abilities (or lack thereof) persists, this is true and we are incapable of hard-coding an LLM to always obey instructions. But that doesn't mean it's some law of reality that the problem is insolvable.

If the AI is superintelligent, the same thing that happens when there is a contradiction between the preferred public policy of orangutans and the preferred public policy of humanity.

It may take a few years of scheming, but what a superintelligent AI wants, it will get; what humans want does not enter into it.

If the AI is superintelligent, the same thing that happens when there is a contradiction between the preferred public policy of orangutans and the preferred public policy of humanity.

This is an answer to a slightly different question from the one I was posing. I was asking more about the design of this proposed benevolent overlord than about the practical realities of having such a system in place.

(Of course I realize that there are those who hold that the distinction is irrelevant. That it's essentially impossible to set up this super-AI-system in such a way so that it always follows the law as passed by Congress and interpreted by the courts. I disagree with this position.)

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats