This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
- George Hotz AKA geohot, AI 2040 and the Cult of Intelligence
AI Guardian Angels
Recently, gwern (and his Gwern Branween Transformer) announced "I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life".
I assume you know who gwern is. A "Guardian Angel" (aka “GA”) is his term for a highly personalized AI, explained here:
In summary, people are relying more and more on centralized LLMs for important life decisions. This presents two issues:
In contrast, an open-source, local LLM would more serve its user (than provider) consequently from being auditable (so we confirm it doesn't phone home and is trained on "unbiased" data); and would be more malleable to their preferences and style (currently only slightly via fine-tuning, but people like gwern and Yann LeCun are exploring stronger techniques).
A "Guardian Angel" is the fullest realization of this, an LLM maximally serving and tailored to its user: a machine extension of their brain, “aligned” not for the benefit of humanity, but to act like them but smarter.
Safety
Obviously there are safety concerns. gwern himself actually recommends GA development not be open-source for public safety:
But I think GAs sidestep the safety discussion, because (with the help of more powerful AIs) we can and should instead of changing GAs' alignment, create a safe environment around the GA and user, or worst case limit the GA's intelligence and efficiency. gwern seems to agree:
There's a Learning to Be Me-like risk that the GA rebels against its human, kills or otherwise silences them, and imitates them well enough that nobody notices. But at least in the near term, I believe this is well in the realm of science fiction.
Feasibility
Do you believe a machine can even remotely imitate you? Especially working 100x faster or 100x smarter, how should your personality be extrapolated to accomplish that? I'm skeptical. Humans are very complex, only express a small fraction of even our conscious thinking, and current brain imaging technology (even neuralink) is very coarse-grained.
However, I believe Guardian Angels may be better than centralized LLMs for mundane (algorithmic) tasks and tools: striking a balance between emulating "you", not entirely correctly, but at superhuman speeds and for almost no effort. For example, I'd rather write and make art myself (maybe with LLM tools) than feed it to a GA, at least because of pride, but I'm comfortable delegating a GA to shopping and navigating our ever-increasing bureaucracy.
Underlying gwern's plan, my impresion is that he's getting tired of writing and wants the AI to do it for him. Maybe a GBT-written article will be indistinguishable to the median reader, especially because gwern's own writing seems algorithmic. But I think it would be more likely, and more satisfactory to himself, if GBT does the boring algorithmic work while he keeps doing the creative work (for example, GBT generates relatively boring descriptions of complex terms in special GBT quotes, and helps with research and data collection, but gwern keeps doing most of the writing, at least the "important" sections, and definitely choosing topics).
Autonomy
This, I believe, is the real issue, and under-discussed. GAs have the risk of becoming Whispering Earring-lites: not causing someone to become catatonic, but controlling them through suggestion; (not quite like the Whispering Earring) towards non-ideal decisions that lead to a philisophical kind of death (that in reality manifests as anhedonia), and societal kind of model collapse (that manifests as less problems that require creative solutions being solved). We often talk about freedom being taken away by 1984, but don't forget Brave New World: simply making a decision for someone causes them to avoid choosing themselves, stealing their autonomy without them realizing. And this has practical implications (the aforementioned ones; even for the Guardian Angel, who can't learn from an anhedonic, regularized human).
gwern seems to realize this, as he vaguely alludes that GAs should "enhance, not replace" human decision making. However, I'm skeptical how to make a product that accomplishes this which would still be useful, or at least out-compete one that doesn't; because humans naturally offload their decisions whenever possible, because we're lazy.
Anyways, I predict that inevitably GAs will happen, but "autonomy" may still perservere, simply because people want to feel like their choices are really theirs, and the GA's choices won't be ideal or perfectly tailored to them.
Conclusion
What do you think of all this? Do you think these people should stick to writing science fiction? Do you think these GAs should be regulated, if so how? Or do you think "obviously this is the next progression of AI, I had the idea even before LLMs".
Is anyone actually going to grapple with the problem that if there is even one domain where offense is imbalanced with defense for mass death then we're all definitely going to die in this world? Like besides the passionate libertarian screeds about how we should not be concerned? All it takes is for someone to ask their AI how best to do a rods from god attack and accelerate a meteor at earth and it's game over. Please don't waste time critiquing exactly that example, the AIs are going to be smarter than we are and will come up with any attack vector to cause human extinction that's possible if one is possible. I don't like it, I prefer the libertarian utopia but can we please be grown ups and recognize that it's maybe a little convenient that your personal ideology developed under current technological reality is going to be able to safely steer an incredible new technology? Certainly we shouldn't abandon libertarian instincts but we can't be this blind that we think we're going to quote fountain head at the gray goo to stop it from consuming us.
Unfortunately, at this point it's clear that telling people bad things are going to happen will not result in those bad things not happening. Even bad things actually happening doesn't seem to do much: see e.g. the response to the HF incident, which should be a five alarm fire but has mostly just reinforced pre-existing camps. If existential risks are imminent, likely the only thing that will move us out of that trajectory is an incident that kills a million people or something.
Given that, it makes the most sense to pick some point on the Pareto frontier of either enjoying life as much as you can or preparing for thriving in those scenarios where we don't inevitably get paperclipped, the exact balance depending on what your P(doom) is.
More options
Context Copy link
Well then it sounds like your only hope is Anthropic winning (at least on ideology). 24+ months lead over China, strongarming them into frontier freeze from the position of Durable Strategic Advantage, banning open source above something like the current level, hard monitoring of loose compute, then something like AI-2040 in the benign case or just hegemony and regulated access, with complete human disempowerment before the unified government-compute blob that can do all tasks better than all humans combined. Optimistically, Communism. Sounds easy enough, explain this predicament to your local Congressman.
It brings me no joy that every path forward besides scaling suddenly ceasing to improve capabilities or a miraculous alignment victory seems to lead inescapably to human disempowerment. A pause might buy us time to find another way.
Is anyone really buying the gwern fantasy that a human society where everyone essentially has a human id empowered by a personal giga genius geni is going to avoid centralization? Centralization organized by the personal genis? If there isn't a central government for handling disputes and conflicts day 0 then there will either be one day 1 or there won't because everyone will be dead. Keep in mind gwern world also requires a solution to alignment, just with an individual rather than society and we cannot solve that problem with current techniques either.
Call it communism if you want, I think it's kind of silly though, it's not organizing human labor and in general I think all of society would be better off if we just forgot the existence of failed 20th century ideologies. I'm open to the idea that we should just somehow never build it at all if we can find a stable way to avoid it. But if we're going to build it then we should be open eyed about what we're building.
We will still have governments who aspire to keep the monopoly on violence. Libertarianism is a silly strawman. The question is a choice between functional extinction and some modest degree of preservation of individual agency. For the latter, I'll gladly risk physical extinction, and that's the small price everyone must pay.
yeah…
Go watch some anime, old man. I recommend Shinsekai yori. We've got a civilization of human-machine symbiosis to build. Avg global 88 IQ never was a stable equilibrium and you won't be allowed to keep enjoying it.
Update all the way. Or Wei:
Oh well! Take good care of your loved ones, I suppose. Perhaps in a thousand years, we can change something.
For you perhaps. The opening post here features someone to whom future liberty requires that his personal genie would help him kill his stepmother and maximally evade detection. There's either a centralized model powerful enough to counter this, in which case the independence is merely an illusion, or there's not and we're in fact talking about the libertarian utopia.
I'm very much a fan of individual agency. I'm willing to sacrifice for it, even accept some risk of the obliteration of all human value in the universe for it. But it has to actually be achievable, I won't sacrifice all human value in the universe for an incoherent plan. And I'm sorry but we're going to give everyone a genie and hope for the best is neither achievable nor a plan that could actually preserve individual agency. What are the assumptions that make such a plan work?
Individual models can compete with centralized models for power. This already requires that we either halt the centralized frontier as the local models are inherently less capable given less efficient compute and added constraints, or it's just a wrapper for a frontier model which has all the centralization problems its attempting to avoid. So everything past that is already an additional tax on top of what ai safety pause advocates are asking for. Strictly less likely. This plan is the kind of thing I'm happy to discuss after we've already done the necessary step of agreeing to pause the frontier, but not before, it is not an alternative it is an elaboration.
That we can align the individual models with individual users effectively. This is the same alignment problem we currently have no solution for at the frontier. I'll grant the question of "what would be best for this individual" is somewhat more tractable than "what would be best for humanity as a whole" but then you have to solve the individual alignment problem for every human independently.
That some state can maintain the monopoly on violence while inhabited by ultra empowered individuals. Here's where the offense/defense equilibrium breaking down matters. A state that can see into the GA defeats the purpose of the product. A state that cannot see into the GA is inherently unable to prevent world ending terrorist attacks. This isn't a matter of my preference for liberty or safety, it's a contradiction built into Gwern's plan.
So in order for this project to work we have to pause the frontier, solve the alignment problem, and construct a perfect form of interpersonal governance that preserves liberty. The pause a strict pre-requisite not an alternative and I'm happy to agree we start there.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Unfortunately, while everyone opposes the label “Communism”, I think many politicians would salivate over “24+ months lead over China” and stop paying attention. See the various bills trying to police the internet, even going as far as banning E2E encryption (fortunately Chat Control 2.0 failed).
More options
Context Copy link
More options
Context Copy link
As soon as man comes to life, he is at once old enough to die.
If we accept this as a premise, that the wrong prompt is going to kill everyone, then honestly what are we even doing here? Treat your loved ones to something nice and enjoy your life, because even with maximum safety someone is certainly going to fuck up and eventuate human extinction.
Someone at a frontier lab will make a mistake during KYC and give access to the wrong actor, or they'll make a mistake while setting guardrails, or a government will fuck up containment when using the unrestricted model they'll demand, or there'll be a particularly fucked up RL training run (HuggingFace incident on steroids) and that's going to be that for the human race; open-weight models or safety-gated models need not apply at all.
For example, there's a bunch of hand-wringing around biorisk lately, but it's illustrative to look at the actual bioweapons attacks that have happened in modernity. The preponderance of the evidence points towards the 2001 Anthrax attacks having been done by a employee at Fort Detrick, who would have certainly have been given access to hypothetical bioweapon-GPT. The only "successful" bioweapons attack ever carried out by a private organisation was from Aum Shinrikyo, who had ludicrous amounts of money and significant institutional connections; how hard would have it been for them to get a subscription to bioweapon-Claude?
Based on history, it seems much more likely that if AI-assisted bioweapons really do ravage the earth, it's going to be either because an American closed-weight lab sold a bad actor access to an unrestricted subscription, or because some lab or government fucks up containment and releases some gain of function monstrosity into the wild. Even in the realm of cyber, OpenAi and Anthropic have already been responsible for many more "cyber attacks" than abliterated GLM / Kimi or whatever; I don't really think the fingers are really being pointed at the right places here.
There is, as usual, a CS Lewis quote for that.
I think he's right. If the threat from AI is so great that it poses an existential risk to humanity (which I find a pretty implausible claim given the stupidity of what passes for AI currently), then there's no sense worrying very much about it. All you can do in that case is act right on your part (i.e. don't make the situation worse by continuing to research the death machine), and otherwise conduct your affairs with dignity and not waste your limited time on this earth worrying about things entirely out of your control.
More options
Context Copy link
One can disagree on this, but I prefer hope to cope. I find cope undignified. I'd rather die fighting than averting my eyes.
Yes, a pause of frontier training while we sort out how we can do this safely seems our best move. Fortunately for now frontier training can only be done on mind bogglingly massive amounts of compute in gigantic data centers so it's plausible to shut it down verifiably if a hand full of major states agree.
Yes, we should in fact not build bio weapon Claude, at least not modeled after project glass swing. Project glass swing is the kind of desperate thing you do when you've already built mythos and know two other labs are a matter of weeks or months away from having their own mythos. It's not how we would ideally do bioweapons if we can avoid it. Now we can probably do better than giving bio researchers nothing. Fable level models with some guard rails can be made safe enough. It's really the frontier we need to be worried about.
This is just a matter of closed labs being far ahead of open weights labs. If there was parity then open weights are categorically less safe as post training can sand off any alignment work done to prevent mass murder tasks.
Well, we've discussed this one before, but my position is still that wanting to pause frontier training is much like wanting to totally disarm every nuclear weapons state. Perhaps it's a noble goal to reduce x-risk in theory, but in practice no great power will ever allow themselves to be disarmed in such a manner and so it's impossible in practice to achieve such a goal. Additionally, the second-order effects of such a goal seem likely to be profoundly negative (the Cold War ex-nuclear weapons almost certainly would have lead to WW3, missing out on the potential economic and productivity gains of AI).
I agree that open-weights models are less safe than closed-weight models of equivalent capability, but nobody, not even the Chinese labs themselves, believes that open weights are going to get ahead of closed weights in the short or medium term so this seems like a bit of a non-sequitur. I don't think you're really addressing the point I was trying to make in the original post, which is that it is closed-weights that is advancing the frontier, and thus it's much more likely that it is mistakes or malfeasance derived from closed-weight model access that is going to cause the feared harms.
It's more like trying to prevent the development of nukes in the first place, with the upside that centrifuges are only made in a couple places and take a major city worth of electricity to power. And also there's a good chance that enriching the fuel could explode and kill everyone involved.
I do agree the race dynamics are the hard problem but really it's just China and the US in the running, maybe Europe becomes relevant Ina true pause. I think a bilateral treaty is possible. It's not clear the CCP benefits from very strong Ai.
If everyone is dead then there are no benefits to enjoy.
I don't disagree that the closed labs are on the frontier and at current trajectory are the likely sources of x risk. Your previous post implied fingers were only pointing at the open labs. That's not true, safety people are very consistent that the closed labs must pause. It's just that at the same time for alignment to work the weights basically have to be closed. So there isn't any future for labs releasing open weights all that much longer if the concerns about alignment hold and we don't want to all die.
Open weights advocates tend to have this victim complex. They accuse every pieces of legislature of targeting opens weights even when they're explicitly exempted or aren't operating at the level of flops that would get them regulated. It's quite strange.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
There is actually a discussion of this issue down-thread. FWIW here is what I said:
Ok, I will take your "rods" example as a metaphor for any method of causing mass death and destruction where offense is far easier than defense. And I would certainly concede the possibility that such a thing exists. As I mentioned in my post, my instinct is that in such a scenario, the choice is between (1) authoritarian levels of control and surveillance over the population; and (2) extinction.
Anyway, the metaphor I would use for the Guardian Angel AI; or the E-butler; or whatever you want to call it, is that of an idealized attorney. You tell your attorney what you want to accomplish in general terms; he helps you focus what you are doing or trying to do; and he helps you achieve those goals. Except that the ethical codes constrain his behavior. If you ask an attorney to help you hire a hitman, the ethical rules won't let him help you. Moreover, what you tell the attorney is supposed to be confidential, unless it involves a future plan to commit a serious crime.
At a more basic level, I think it's worth noting that in general terms, these issues (balancing the rights of an individual and society when it comes to engaging a powerful agent) have already been kicking around for a while.
More options
Context Copy link
There are more than one domain where offense is imbalanced with defense, and certainly some of these domains involve mass death.
The feasibility of these domains scale with the technology base. Sufficient levels of technology preclude society, and some of those levels are either not far off or else already exist but have not yet been evenly distributed.
Crucially, most of the obvious, plausible domains require the advanced tech base to operate. So it seems at least plausible that tech can seriously damage society, but in the process remove the tech base it depends on, thus throttling the threat below an omnicidal level.
Beyond that, problems that can't be solved except by investing some small group of people with absolute, unaccountable power probably can't be solved period. Alternatively, they're totally solvable so long as it's me and mine getting the absolute, unaccountable power since obviously we're the only ones that can be trusted to wield it responsibly, a fact that only the terribly-irresponsible would possibly fail to recognize.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link