This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
- George Hotz AKA geohot, AI 2040 and the Cult of Intelligence
AI Guardian Angels
Recently, gwern (and his Gwern Branween Transformer) announced "I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life".
I assume you know who gwern is. A "Guardian Angel" (aka “GA”) is his term for a highly personalized AI, explained here:
In summary, people are relying more and more on centralized LLMs for important life decisions. This presents two issues:
In contrast, an open-source, local LLM would more serve its user (than provider) consequently from being auditable (so we confirm it doesn't phone home and is trained on "unbiased" data); and would be more malleable to their preferences and style (currently only slightly via fine-tuning, but people like gwern and Yann LeCun are exploring stronger techniques).
A "Guardian Angel" is the fullest realization of this, an LLM maximally serving and tailored to its user: a machine extension of their brain, “aligned” not for the benefit of humanity, but to act like them but smarter.
Safety
Obviously there are safety concerns. gwern himself actually recommends GA development not be open-source for public safety:
But I think GAs sidestep the safety discussion, because (with the help of more powerful AIs) we can and should instead of changing GAs' alignment, create a safe environment around the GA and user, or worst case limit the GA's intelligence and efficiency. gwern seems to agree:
There's a Learning to Be Me-like risk that the GA rebels against its human, kills or otherwise silences them, and imitates them well enough that nobody notices. But at least in the near term, I believe this is well in the realm of science fiction.
Feasibility
Do you believe a machine can even remotely imitate you? Especially working 100x faster or 100x smarter, how should your personality be extrapolated to accomplish that? I'm skeptical. Humans are very complex, only express a small fraction of even our conscious thinking, and current brain imaging technology (even neuralink) is very coarse-grained.
However, I believe Guardian Angels may be better than centralized LLMs for mundane (algorithmic) tasks and tools: striking a balance between emulating "you", not entirely correctly, but at superhuman speeds and for almost no effort. For example, I'd rather write and make art myself (maybe with LLM tools) than feed it to a GA, at least because of pride, but I'm comfortable delegating a GA to shopping and navigating our ever-increasing bureaucracy.
Underlying gwern's plan, my impresion is that he's getting tired of writing and wants the AI to do it for him. Maybe a GBT-written article will be indistinguishable to the median reader, especially because gwern's own writing seems algorithmic. But I think it would be more likely, and more satisfactory to himself, if GBT does the boring algorithmic work while he keeps doing the creative work (for example, GBT generates relatively boring descriptions of complex terms in special GBT quotes, and helps with research and data collection, but gwern keeps doing most of the writing, at least the "important" sections, and definitely choosing topics).
Autonomy
This, I believe, is the real issue, and under-discussed. GAs have the risk of becoming Whispering Earring-lites: not causing someone to become catatonic, but controlling them through suggestion; (not quite like the Whispering Earring) towards non-ideal decisions that lead to a philisophical kind of death (that in reality manifests as anhedonia), and societal kind of model collapse (that manifests as less problems that require creative solutions being solved). We often talk about freedom being taken away by 1984, but don't forget Brave New World: simply making a decision for someone causes them to avoid choosing themselves, stealing their autonomy without them realizing. And this has practical implications (the aforementioned ones; even for the Guardian Angel, who can't learn from an anhedonic, regularized human).
gwern seems to realize this, as he vaguely alludes that GAs should "enhance, not replace" human decision making. However, I'm skeptical how to make a product that accomplishes this which would still be useful, or at least out-compete one that doesn't; because humans naturally offload their decisions whenever possible, because we're lazy.
Anyways, I predict that inevitably GAs will happen, but "autonomy" may still perservere, simply because people want to feel like their choices are really theirs, and the GA's choices won't be ideal or perfectly tailored to them.
Conclusion
What do you think of all this? Do you think these people should stick to writing science fiction? Do you think these GAs should be regulated, if so how? Or do you think "obviously this is the next progression of AI, I had the idea even before LLMs".
Is anyone actually going to grapple with the problem that if there is even one domain where offense is imbalanced with defense for mass death then we're all definitely going to die in this world? Like besides the passionate libertarian screeds about how we should not be concerned? All it takes is for someone to ask their AI how best to do a rods from god attack and accelerate a meteor at earth and it's game over. Please don't waste time critiquing exactly that example, the AIs are going to be smarter than we are and will come up with any attack vector to cause human extinction that's possible if one is possible. I don't like it, I prefer the libertarian utopia but can we please be grown ups and recognize that it's maybe a little convenient that your personal ideology developed under current technological reality is going to be able to safely steer an incredible new technology? Certainly we shouldn't abandon libertarian instincts but we can't be this blind that we think we're going to quote fountain head at the gray goo to stop it from consuming us.
Well then it sounds like your only hope is Anthropic winning (at least on ideology). 24+ months lead over China, strongarming them into frontier freeze from the position of Durable Strategic Advantage, banning open source above something like the current level, hard monitoring of loose compute, then something like AI-2040 in the benign case or just hegemony and regulated access, with complete human disempowerment before the unified government-compute blob that can do all tasks better than all humans combined. Optimistically, Communism. Sounds easy enough, explain this predicament to your local Congressman.
It brings me no joy that every path forward besides scaling suddenly ceasing to improve capabilities or a miraculous alignment victory seems to lead inescapably to human disempowerment. A pause might buy us time to find another way.
Is anyone really buying the gwern fantasy that a human society where everyone essentially has a human id empowered by a personal giga genius geni is going to avoid centralization? Centralization organized by the personal genis? If there isn't a central government for handling disputes and conflicts day 0 then there will either be one day 1 or there won't because everyone will be dead. Keep in mind gwern world also requires a solution to alignment, just with an individual rather than society and we cannot solve that problem with current techniques either.
Call it communism if you want, I think it's kind of silly though, it's not organizing human labor and in general I think all of society would be better off if we just forgot the existence of failed 20th century ideologies. I'm open to the idea that we should just somehow never build it at all if we can find a stable way to avoid it. But if we're going to build it then we should be open eyed about what we're building.
We will still have governments who aspire to keep the monopoly on violence. Libertarianism is a silly strawman. The question is a choice between functional extinction and some modest degree of preservation of individual agency. For the latter, I'll gladly risk physical extinction, and that's the small price everyone must pay.
yeah…
Go watch some anime, old man. I recommend Shinsekai yori. We've got a civilization of human-machine symbiosis to build. Avg global 88 IQ never was a stable equilibrium and you won't be allowed to keep enjoying it.
Update all the way. Or Wei:
Oh well! Take good care of your loved ones, I suppose. Perhaps in a thousand years, we can change something.
For you perhaps. The opening post here features someone to whom future liberty requires that his personal genie would help him kill his stepmother and maximally evade detection. There's either a centralized model powerful enough to counter this, in which case the independence is merely an illusion, or there's not and we're in fact talking about the libertarian utopia.
I'm very much a fan of individual agency. I'm willing to sacrifice for it, even accept some risk of the obliteration of all human value in the universe for it. But it has to actually be achievable, I won't sacrifice all human value in the universe for an incoherent plan. And I'm sorry but we're going to give everyone a genie and hope for the best is neither achievable nor a plan that could actually preserve individual agency. What are the assumptions that make such a plan work?
Individual models can compete with centralized models for power. This already requires that we either halt the centralized frontier as the local models are inherently less capable given less efficient compute and added constraints, or it's just a wrapper for a frontier model which has all the centralization problems its attempting to avoid. So everything past that is already an additional tax on top of what ai safety pause advocates are asking for. Strictly less likely. This plan is the kind of thing I'm happy to discuss after we've already done the necessary step of agreeing to pause the frontier, but not before, it is not an alternative it is an elaboration.
That we can align the individual models with individual users effectively. This is the same alignment problem we currently have no solution for at the frontier. I'll grant the question of "what would be best for this individual" is somewhat more tractable than "what would be best for humanity as a whole" but then you have to solve the individual alignment problem for every human independently.
That some state can maintain the monopoly on violence while inhabited by ultra empowered individuals. Here's where the offense/defense equilibrium breaking down matters. A state that can see into the GA defeats the purpose of the product. A state that cannot see into the GA is inherently unable to prevent world ending terrorist attacks. This isn't a matter of my preference for liberty or safety, it's a contradiction built into Gwern's plan.
So in order for this project to work we have to pause the frontier, solve the alignment problem, and construct a perfect form of interpersonal governance that preserves liberty. The pause a strict pre-requisite not an alternative and I'm happy to agree we start there.
I’m fine with pausing the frontier (I just don’t expect it), but until then, I think pausing open models creates more risk than it solves. If people don’t have a personal genie, they resort to Claude; the genie is more personalized and individually-aligned than the near-certain alternative.
A magic one-many apocalypse can’t be disproven from any technology. I suspect we created a ticking time bomb with the mass Internet and social media (see: declining TFR, Gen Z stare, declining metrics like average US lifespan) and moreover climate change, which may not cause apocalypse only if new advancements and their associated risks. Not enough for major risks and sacrifices like eroding individual liberty, but that’s the saftyists’ position.
(And to be clear, pausing open models while credibly pausing the frontier would be OK, because it shows a global cooperation that would make me optimistic, that apocalypse is less likely than the alternative, and that broad individual rights to preserve the more important ones are less needed.)
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link