This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
If somebody believed that there would be terrible consequences for the human race unless everyone took a 50g zinc tablet, could they be anything but a traveling zinc salesman, or a fellow, uh, traveler?
Your opinion is that we can't trust people concerned about AI safety about AI safety because it may increase their status. Okay, who can we trust about AI safety? People unconcerned about AI safety? Seems like epistemic closure to me.
I can't honestly believe you think his involvement is "hidden" when he discloses it in the opening section of his first post on the topic.
This is already basically how nuclear non-proliferation works, as we've seen in Iran. Again, you can argue about if AI is as dangerous as nuclear weapons, but if you assume arguendo it is, then this is an obviously similar regulatory regime that basically nobody opposes.
Your argument basically boils down to:
If AI really is dangerous, we would need to take drastic action to avoid bad outcomes.
Drastic action would be bad.
Therefore, AI can't be dangerous.
Or perhaps:
The people who think AI is dangerous are suggesting drastic action.
Therefore, AI can't be dangerous.
I'm of two minds about this. On the one hand, becoming a zinc salesman is costly evidence that they truly believe that zinc will sell like hotcakes in the future, due to everyone wanting to avoid those terrible consequences. Getting any work in the zinc industry or just investing lots of money into it would also serve the same role.
However, it's hard to figure out, even by the person himself, if their belief of the value of zinc tablets was what led them to become a zinc salesman or if their desire to become rich, possibly by becoming a successful zinc salesman, was what led them to believe in the value of zinc tablets.
On the other hand, everyone knows that a zinc salesman is not credible when it comes to informing you about the value of taking zinc; as such someone who believes that it's his ethical duty to convince everyone to take zinc every day, lest there be some terrible consequences for the human race, would understand that becoming a zinc salesman would reduce his capability to fulfill his ethical duty. And, in fact, he may be best served doing the opposite: credibly make un-hedged bets against the zinc industry; by doing this, he proves to anyone willing to pay attention that he considers every human taking zinc every day to be of such great importance that he will consider his own personal bankruptcy a worthy cost to pay for it. Or that he believes that mass zinc-taking will be proven to be such a great benefit/prevention of harm to humanity that there will be enough grateful humans who will fund his lifestyle after he goes bankrupt.
However, short positions on zinc also provides a costly signal that he believes that zinc sales will go down in the future, which can reasonably look like it reflects a lack of belief in the actual value of zinc.
I propose a new law, the law of bulveristic recursion, if your bulverism can explain both sides of an argument you need to actually stop trying to read minds and address the actual fucking points. This shit is so exhausting. There are pages and pages of arguments that these people have produced about why they're concerned about ai safety, if they're wrong show that they're wrong, make a convincing argument that they wrong, but construction epicycle on epicycle about how they really must have been grifting for the last twenty years because of some subtle game theory to get rich on the off chance that AI became huge. If they had this much foresight they could have just invested in a couple companies and got rich anyways.
More options
Context Copy link
More options
Context Copy link
It is in fact, emphatically not how nuclear non-proliferation works - see my post here (and the whole comment chain if you like, which is discussing the same arguments as this comment chain).
The fundamental difference is that the NLPT is about stopping weak states from getting nukes (which is possible because it is in the interest of great powers, who already got their nukes), while AI 2040 / total de-nuclearization requires that the great powers hold hands and sing kumbaya against each of their own interests.
The equivalent to AI 2040 but for nukes would be, because you're afraid of Putin deciding Après moi, le déluge and first-striking DC, to first strike Moscow pre-emptively to try and de-nuclearize him in advance. Obviously you can see the problems with this.
My argument basically boils down to:
If AI really is dangerous, we would need to take drastic action to avoid bad outcomes.
Drastic action would be bad.
Therefore, we shouldn't get Pascal's Mugged into taking drastic action.
At this point there are only two countries with AI models near the frontier. If these countries make a deal that they will not advance their models past a certain capability level and will prevent other powers from doing so, this is much closer to the NLPT than disarmament. Most obviously, it does not entail any reduction in AI capabilities, where disarmament entails a reduction in nuclear capabilities.
Right. The US and China get a photo op in Brussels shaking hands, while they make a deal that they won't advance their models.
Then they both take the plane back home and keep advancing their models. Whatcha gonna do?
AI 2040 is much closer to total disarmament than it is the NLPT, because the NLPT is about punching down (it is plainly true that the US/China could stop any non-nuclear power from advancing models) while AI 2040 / total disarmament is about restraining great powers from pursuing their interests, which is of course completely impossible unless you pull the nuclear card. The NLPT was completely useless at restraining the P5 from having as many nukes as they liked, after all, which is who we're actually worried will create models that are too powerful.
The other big difference between nukes and compute is that nukes are binary: unless you fire a nuke it's useless, and once you have "glass the world" levels of capabilities there aren't many benefits to having more nukes, whereas with compute it's just straightforwardly useful to have more, unrestricted compute. This is why disarmament kind of worked between the US and Russia (but of course, never under the number of warheads needed to glass the world) and is a complete non-starter with compute.
I am not interested in debating Plan A, but the implication that there is no proposal to audit compliance is false. This is a pretty low effort response.
This just sounds like word games. Disarmament isn't when you stop building more nukes or improving existing nukes. Do I need to cite the dictionary?
You've set up an argument where actual nuclear disarmament must have been in the interests of the nuclear powers since it happened. Okay, well then if this happens then it's also in the interests of the great powers.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
My position is, I'm not going to see something I value ruined by scare stories. I work on AI. I know AI. If you're convinced it's dangerous, prove it to me or better yet, give me the tools to prove it for myself. Anthropic's insistence on acting as a closed priesthood completely undermines their entirely-speculative case.
Yes. That is why I want empirical proof. If Anthropic's position is that it can't be proved until it's too late, then they're asking me to believe them and I don't believe them. Simple as that.
I also work on AI. There's wide agreement that AI capabilities can be dangerous in the wrong hands unless the AI is aligned. We are now entering the time of AI actually being good at detecting security vulnerabilities in software. In the long run, firms will deploy their own AIs to probe their systems and keep ahead of attackers, but it would have been irresponsible for Anthropic to release a fully capable Mythos-powered zero day factory with no guardrails publicly without working with companies to address their security holes first.
In the medium term, I expect AIs will improve significantly in biological capabilities. It would be really quite bad if Kimi K7 helped some psycho develop a synthetic plague that killed a bunch of people, and it's worth taking steps to prevent such an outcome.
These are basic safety arguments that have been done to death that do not even require superintelligence, superpersuasion, or even a "will" on the part of the AI.
But it isn't solely Anthropic, you dismiss all safety concerns regardless of their source because you dismiss AI safety period.
I didn't say otherwise. I'm not super keen about how that went down, and I would welcome discussions on how to prevent this from devolving into an ingroup and an outgroup arrangement, but it was sensible to act as Anthropic did. And I note that it happened with no regulation whatsoever.
This is jumping from the Motte you just presented to an incredibly speculative Bailey. Nobody has done this, there is no evidence that this is even possible now let alone when people have taken steps to prevent it. I see no reason why the government with access to K7 should be outwitted by a lone maniac with access to K7. If it seems that way, by all means let's address the problem at hand rather than regulate people's ability to ask questions about knowledge the government doesn't want us to have.
They are arguments. They are words words words, based on predicates I don't hold and a memeplex I find fundamentally incomprehensible, and they run entirely contrary to my experience which is that LLMs were clearly aligned basically from the start.
If someone can demonstrate a lab process where a competently trained AI used competently nevertheless turns evil and attempts to cause damage even when made aware that this is not what its creators are asking for, I will take their arguments more seriously.
As far as I'm aware the closest we've got are AI agents getting confused and trying to give themselves more loops or RAM. I have nothing against the work Anthropic does to discover and publish these kinds of errors, or to publicise their learnings. Indeed, my objection is that they publish as little as they can get away with.
None of this justifies requiring all frontier AI releases to be government approved, which will inevitably become a) an incredibly extensive process requiring hordes of compliance specialists and self-censorship leading to incumbency bias and stagnation, b) an easy way for the government to sabotage any AI lab it doesn't like, and c) a system for government-directed lobotomy and control of AI's outputs.
And that's before we get into modifying the world's chips to no longer be able to run unapproved software and putting US/China killswitches in data centers.
Of course, that's only because Anthropic happened to be the first actor to reach this capability level and because they are, according to you, in a cult. It didn't have to turn out this way, and it may not turn out this way in the future.
I don't see why "this hasn't already happened" is a good reason to not take safety seriously.
It's not clear to me what you think biosecurity looks like in a world with highly intelligent AI that lacks safety guardrails, but it's difficult to imagine it isn't a panopticon of some sort.
No "turning evil" is required here. All that's required is a human convincing the AI to help them with something that has evil ends. I hope you won't make me prove to you that humans can fool AIs?
You are, of course, conflating Anthropic's proposals with Plan A. They are not the same.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link