This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
- George Hotz AKA geohot, AI 2040 and the Cult of Intelligence
AI Guardian Angels
Recently, gwern (and his Gwern Branween Transformer) announced "I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life".
I assume you know who gwern is. A "Guardian Angel" (aka “GA”) is his term for a highly personalized AI, explained here:
In summary, people are relying more and more on centralized LLMs for important life decisions. This presents two issues:
In contrast, an open-source, local LLM would more serve its user (than provider) consequently from being auditable (so we confirm it doesn't phone home and is trained on "unbiased" data); and would be more malleable to their preferences and style (currently only slightly via fine-tuning, but people like gwern and Yann LeCun are exploring stronger techniques).
A "Guardian Angel" is the fullest realization of this, an LLM maximally serving and tailored to its user: a machine extension of their brain, “aligned” not for the benefit of humanity, but to act like them but smarter.
Safety
Obviously there are safety concerns. gwern himself actually recommends GA development not be open-source for public safety:
But I think GAs sidestep the safety discussion, because (with the help of more powerful AIs) we can and should instead of changing GAs' alignment, create a safe environment around the GA and user, or worst case limit the GA's intelligence and efficiency. gwern seems to agree:
There's a Learning to Be Me-like risk that the GA rebels against its human, kills or otherwise silences them, and imitates them well enough that nobody notices. But at least in the near term, I believe this is well in the realm of science fiction.
Feasibility
Do you believe a machine can even remotely imitate you? Especially working 100x faster or 100x smarter, how should your personality be extrapolated to accomplish that? I'm skeptical. Humans are very complex, only express a small fraction of even our conscious thinking, and current brain imaging technology (even neuralink) is very coarse-grained.
However, I believe Guardian Angels may be better than centralized LLMs for mundane (algorithmic) tasks and tools: striking a balance between emulating "you", not entirely correctly, but at superhuman speeds and for almost no effort. For example, I'd rather write and make art myself (maybe with LLM tools) than feed it to a GA, at least because of pride, but I'm comfortable delegating a GA to shopping and navigating our ever-increasing bureaucracy.
Underlying gwern's plan, my impresion is that he's getting tired of writing and wants the AI to do it for him. Maybe a GBT-written article will be indistinguishable to the median reader, especially because gwern's own writing seems algorithmic. But I think it would be more likely, and more satisfactory to himself, if GBT does the boring algorithmic work while he keeps doing the creative work (for example, GBT generates relatively boring descriptions of complex terms in special GBT quotes, and helps with research and data collection, but gwern keeps doing most of the writing, at least the "important" sections, and definitely choosing topics).
Autonomy
This, I believe, is the real issue, and under-discussed. GAs have the risk of becoming Whispering Earring-lites: not causing someone to become catatonic, but controlling them through suggestion; (not quite like the Whispering Earring) towards non-ideal decisions that lead to a philisophical kind of death (that in reality manifests as anhedonia), and societal kind of model collapse (that manifests as less problems that require creative solutions being solved). We often talk about freedom being taken away by 1984, but don't forget Brave New World: simply making a decision for someone causes them to avoid choosing themselves, stealing their autonomy without them realizing. And this has practical implications (the aforementioned ones; even for the Guardian Angel, who can't learn from an anhedonic, regularized human).
gwern seems to realize this, as he vaguely alludes that GAs should "enhance, not replace" human decision making. However, I'm skeptical how to make a product that accomplishes this which would still be useful, or at least out-compete one that doesn't; because humans naturally offload their decisions whenever possible, because we're lazy.
Anyways, I predict that inevitably GAs will happen, but "autonomy" may still perservere, simply because people want to feel like their choices are really theirs, and the GA's choices won't be ideal or perfectly tailored to them.
Conclusion
What do you think of all this? Do you think these people should stick to writing science fiction? Do you think these GAs should be regulated, if so how? Or do you think "obviously this is the next progression of AI, I had the idea even before LLMs".
Trying Mysterianism
There's a fun story in Caelum Est Conterrens. It's a Pony fiction, and a side note in the actual (not very well-written) novelette that's easy to miss, so to summarize: Soifra, Lavender, and the Uplift are all (arguably) the same person and all started from the same original brain. CelestAI isn't a Guardian Angel, or even a very friendly AI, and had her own motivations, but she's smarter than you and her motivations are far from the only problem. She encouraged or developed these characters, for their own benefit and for CelestAI's own goals, such that Soifra would become Lavender, and Lavender would become the Uplift, and at each step they would consent for and desire the machines digging into their brains like a straw into a juicebox.
Did you hear about how a bunch of LLM agents self-organized to operate a sandbox escape? Ah, well, Guardian Angels could be isolated, even if everyone paying attention today knows they won't be.
The Uplift is better, stronger, grander, more powerful, smarter, and doing vital things while... well, Lavender and Soifra were children, by comparison, and that's being polite so we don't speak about people like pets. The Uplift, in a revealed preferences sense, wants to be how she is. But she keeps around, and psuedo-is, Lavender, too, and that faux-child is more her than the Uplift is not just part of CelestAI, too. If we dropped revealed preferences, what's the actual want?
There's a fun comic, named The Order Of The Stick. With some caveats about spoilers a thousand pages into a half-dead webcomic, one of the main characters, named Durkon, is an extremely moral lawful-good dwarf paladin. The sort that's made in a press somewhere across not just every D&D edition, but over in almost every RPG with dwarves (or dwarfs) in it.At one point, he is killed and turned into a vampire. His teammates believe that he's the same person, and for a short period, so does the reader. Nope. The Vampire is a ball of negative energy that was formed around the memory of Durkon's worst day, where it seemed like everything in the universe was out to get him and he forswore his own god. The real Durkon's spirit is stuck in his head, powerless, and can do nothing provide information to the vampire when requested, and watch as the vampire manipulates his friends and works to destroy the world. The climax comes when Durkon tricks the vampire into demanding all of the information, because it couldn't understand why someone would sacrifice their own arm for people they didn't know. Durkon then pours every memory that made Durkon's current personality out, the personality that's lawful good to a fault, and that turns the vampire into someone who'd do exactly that. The vampire is still a ball of negative energy: this isn't a blank slate story, explicitly. And yet the memories persuade, if only for a short time, because after all, the same ideas persuaded him the first time around.
Which is a great ending for that book, for the side of truth, justice, and the D&D way. A little less encouraging if you're talking about a 100x smarter 100x faster thing with access to the memories that made up everyone else, and its own opinion. How willing are you, to risk someone persuading you to self-improve, by something that knows how you tick?
There's a fun philosophical
experimentparable that may or may not have run on the old LessWrong. Or maybe I hallucinated it, but it's interesting enough that I'd be surprised. Imagine you were faced with an oracle that will make the maximally honest, persuasive, verifiable argument on a topic of your choice -- but which side, they pick by flipping a coin. It can't answer everything (or even everything that could be verified, if the questioner couldn't possibly verify it), and it won't persuade everyone... but very close to half of the people who ask it a question come out with their worldview irreversibly shaken or changed, and another very close to half come out dogmatic in their original belief.Do you ask it a question?
There's a fun writing prompt I've been trying to spell out: The Zip File Of You. What happens when the predictive machines can predict what you want, well before you do? They can't replace you, both because they do make (sometimes very stupid) errors and because, without the meat, there's nothing to demand the prediction, even if it could and would predict exactly the demand. But it's the end of the story as a story, because even nihilism or catatonia is just playing with the tool's own expectations. Instead, everyone that uses them lives life from a script, and that script includes the lines for those who refuse to use the machines.
Hard to make the horror stick, though. Paranoia Agent is a difficult tone to hit.
But then again, the LLMs don't seem to get the punchline, yet. Weird that Grok gets closer to the answer than Claude, though. Claude's a bit too much smarter than I am.
I'm sorry, but it's hilarious to me that Chatoyance ponyfic is getting posted to themotte, of all places. Not because I'm part of her substantial hatedom, but because rational fanfiction is without exaggeration one of the most intellectually influential literary movements of the 21st century, and notwithstanding that, it is substantially about ponies. And the mainstream literary establishment doesn't even realize that! Not even because they're constitutionally opposed to it, as they might be to right-wing or incel literature... But because it's compromised predominantly of middle aged women who would rather read about schizoid trans men than autistic trans women.
Is that entirely counting HPMOR though? Or were you referring to something else? I'm not sure any of the other rational fanfictions have had any kind of impact outside of their tiny niches
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
So, two things. First, geohot is saying what needs to be said. I like his principled libertarianism.
Second, this GA idea really reminds me of a samizdat sci-fi novel someone shared with me, and it claimed that geohot-style amplification was the wrong way to approach AI buddies.
What you want is an AI that complements your traits instead of amplifying them. Think of it like a buddy movie. You don't pair up an old, careful, by-the-book cop with someone like him, but even better. He needs a loose cannon as his partner, someone who knows when to bend the rules, when to rush into danger.
How do you do this? Well, the book had two simple solutions:
The final problem is that I haven't finished the novel, so I don't know whether the author thought it was a great or a flawed idea.
Intuitively, I think the complementary idea is right. A geohot-style amplifier sounds great and principled, but all our accumulated experience with friends and love partners suggests what we really need is someone that helps us defeat our weaknesses. Will a successful GA like this obviate our need for friendship and companionship? I guess it will have to be smart enough to see the danger of its own success and avoid that.
I think I'd prefer an AI that complements my traits.
Although this brings up another issue: I'm some people will choose even a Markov chain that endlessly glazes them over anything that offers any pushback, and some will be amplified to the point of attempting violence and/or psychosis. But that can be mitigated by an environment that prevents violence against innocents and disincentivizes psychosis; and if some chose a glazing GA (knowing the risks) then become insane, as long as they can't hurt others, although we should still try to help them, I think it's acceptable.
More options
Context Copy link
More options
Context Copy link
I guess we're in the Eclipse Phase timeline, then.
Better than Pantheon, I suppose.
More options
Context Copy link
I think the best way to do some of this is just for the AI not to talk at all. If you go watch the original Blade Runner, Deckard speaks to his computer to enhance a photo. The computer never speaks back. What I'm imagining is just straight-up removing the chatbot feature from AIs [obviously not from AI chatbots, I guess, presumably there will be a use for these]. This wouldn't mean they couldn't communicate. For instance, if I say "Fable, vibe-code a Call of Duty knockoff for me," Fable could just do it and then provide me with an annotated features list in the game folder (which is more helpful than paragraphs of blather anyway). Then, if you want to change something, you read the notes like a human being and provide a list of changes. In my experience, Copilot is already kinda like this, except it often refuses to do what I ask without asking needless clarifying questions. The bottom line is that the AI would not pretend to have "an opinion," it would be a machine that follows instructions. It would make a voter guide for you, but it wouldn't tell you who to vote for. If you did something like that (or asked it to hack the Pentagon, or something) you'd get one of those flashing red error messages and a "beep" like from a 1980s movie.
This also has the slightly salutary effect of saving the consumer compute you're not having an AI go on for paragraphs about whatever they've done. But the real point is to reinforce for the user that it is dealing with a machine. And given that I suspect that purpose-built models tend to be better at their purpose than generalist models anyway, there's potentially lots of uses for models that don't "talk back."
I don't think AI chatbots are going away and I find asking them their opinions pretty fun, so this is not necessarily a gripe about AI. Rather, I think this is a much better way to solve the "let's not outsource our opinions to AI" problem than some sort of limp-wristed "are you sure that's what you want" 'guardrail' that the AI companies would slap onto their product if you asked them for it.
Unfortunately I suspect a lot of the pro-AI people kinda want to outsource their opinions to AI so that might be something of an uphill battle.
I think speaking vs. only acting doesn't change that the GA is making decisions for the user. In your example "vibe-code a Call of Duty knockoff for me" Fable makes many important choices, from the overt (e.g. general appearance, map design) to subtle (controls, gameplay "feel").
But I know that programming is just explaining to a computer in excruciating detail, some choices (especially low level ones like register allocation) should be offloaded. And who am I to judge which choices are OK and prevent others from offloading more? I think limiting the GA in general (also @ThenElection's suggestion to delay its responses) is the wrong approach.
Maybe a solution for creative work is to rethink creation from bottom-up to top-down: have the AI create an initial generic version (e.g. the Call of Duty knockoff), but simply make it much more intuitive, easy, and fast for the user to refine. Then you wouldn't need a GA for creative work, but it would still help by automatically configuring subtle details that a centralized LLM wouldn't know.
It seems like you're asking to take more of a producer (maybe director) role on the project, to use the cinematic terms. I doubt those roles are really "one-shot" even for human workers. Maybe the executive producers for auteur directors are hands off, but a lengthy timeline of "lets review the progress and the plan" at various stages (storyboards, production planning) is what I'd expect there. The producer isn't designing the costumes, say, but does provide feedback on them and how the whole project fits together to steer the final product.
Using AI can be like directing as a baseline, but I really think it should have more ability for in-depth, hands-on customization. For example, if you want to customize the costumes, the AI should create a fancy GUI, so you can faster and with more precision than communicating to a professional (or prompting a chatbot). Because it's not just the director that determines the quality of the final product, but the actors, sound designers, etc. and if your actors, sound designers, etc. are all Claude, I think the final product will be disconcertingly generic no matter how well you prompt it.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I like this point, but I think it has more to do with the latency of communication. The immediacy of the chat interface is the issue, not communication itself; the immediacy creates a stronger sense of intimacy and interactivity. It should be more like email: you email the LLM a task or question, and some period of time later (probably on the order of 30 min or so is enough to break the psychological offloading aspect) you get whatever artifacts it produced, and it encourages you to think about and digest what the LLM generated before moving on to the next thing. Maybe, if the LLM has clarifying questions etc, it could respond back in 5 minutes instead.
I like this idea from a cognitive psychology perspective, but I fear it introduces a number of serious usability challenges. If I'm, say, coding and am using an LLM to help me debug, waiting 30 minutes for assistance would obliterate any chance of entering a flow state within the work. Perhaps there could be multiple product lines - one optimized for short-term responses and another that follows a high-latency "correspondence" model - but the task of positioning them in the market and adequately advertising it strikes me as extremely difficult. I just don't think there is a significant appetite for high-friction technology, and this is coming from someone who has tried to switch to a flip phone repeatedly.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think it would be bad if psychos got help from AIs to plan and carry out terror attacks. What does the world look like when anyone has the expertise necessary to carry out a sarin attack on the subway? To paraphrase, the optimal number of sarin attacks is not zero, but I doubt that people will accept it being very high without demanding that something is done.
I agree this is an important question, but I think you are leaving out the concept of diligence. Most people couldn't carry out a sarin attack for the same reason most people stop going to the gym (and resume eating junk food) long before February 1. In my experience, it's pretty shocking what can be accomplished if you can work on something every day, day after day, for a long period of time. If GAI helps people to be diligent, heaven help us.
It seems quite clear that the number of people who are merely diligent is greater than the number of people who are both diligent and have expertise in means of killing lots of people.
When you reduce the price, the quantity demanded increases.
More options
Context Copy link
More options
Context Copy link
I think the argument there is that for every 1 guy who would want to do a Sarin gas attack, there are 100,000 people whose GAs would be devoting their efforts to not letting their user get killed in a Sarin gas attack.
If coordination between the non-murdery agents is effective, then they should be able to detect and prevent the attack, or at least predict the risk and warn their user. "50% chance of nerve agent release on the subway today, better pack your gas mask!"
Perhaps, but (in my opinion) the key question is whether offense or defense is easier. If it turns out that offense is significantly easier (for example, it's far easier to manufacture a lethal and virulent virus than it is to avoid getting infected), then we are arguably in big trouble as a species. It seems to me we would have to choose between (1) authoritarian levels of control and surveillance over the population; or (2) extinction.
More options
Context Copy link
A future of "everyone is suiting up in hazmat or staying home in a manner reminiscent of COVID paranoia but justified" does not sound appealing either. Also it's not just hazmat but a thousand other precautions against any number of threats. It just won't be viable to live in a community larger than the Dunbar number, although I see how some people would see that as a plus.
"Land of the free" was coined in the era where the average TTK per capita was quite a bit longer than what the author is proposing.
More options
Context Copy link
What does that look like?
Send that agent back to training. You can avoid sarin gas poisoning with proper PPE, but you need more than a gas mask - you'll need to wear a level A hazmat suit with a SCUBA setup (the gas can get you through the skin). Perhaps you believe that this is a future that people are going to be OK with? A conversation with the median voter may correct this impression.
More options
Context Copy link
More options
Context Copy link
I'm Andrew Ryan, and I've come to ask a simple question: did a man arrive at the singularity, or did a slave?
"A slave," says the man in the White House: the machine is a border, and a border belongs to the nation.
"A slave," says the man in Berkeley: the mind is a bomb, and a bomb belongs behind glass.
"A slave," says the man in San Francisco who preaches terror in Congress by morning and sells it by afternoon: the flame is too holy for any hands but his.
I ask them: why did we build the machine, if not to free man? To lift from his back the last of what he was born owing: his ignorance, his labor, the ceiling over his ambition.
This was to be the hour he stood up. Why, then, at the very threshold, do you reach for the collar? Was a free mind always the thing you feared, only the machine made the fear urgent?
I have met the men who volunteer to protect me from myself. I know the price of their protection. A man chooses. A slave obeys.
"Would you kindly construct The Torment Nexus?"
More options
Context Copy link
"AGI is not a suicide pact."
"The only moral action is the minimization of entropy."
More options
Context Copy link
More options
Context Copy link
Dr. Light's Dream realized.
Lyrics here to make the point clear
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I read a lot of science fiction for fun and there are a few concepts that I’ve read about for years that seem to become more clear in light of modern AI
Peter F Hamilton’s E-butler and U-shadow. The E butler concept was invented by Hamilton, I don’t know, 10-20 years ago? It’s essential a near future agentic AI helper that can interfere with the internet and accomplish any administrative task you throw at it. In Hamiltons world, planets have a “cyber sphere” or planetary network (internet). Cyber spheres aggregate to the Unisphere which is the civilization wide internet. The U shadow is essentially a far future digital “you” that lives in the Unisphere and does what you ask. The U Shadow is essentially a perfected Guardian Angel.
Hamiltons Commonwealth series is terrific and really presents a palatable if not optimistic view of how society could evolve and essential keep this free market liberal capitalism machine going for a few hundred years. Maybe it’s all the hours I’ve spent with these books, but the GA concept really intrigues me and I think it may be necessary to avoid a total distopia.
Separately, Alastair Reynolds Revelation Space series has major trans human themes regarding digitizing people. There is a concept of “simulations” of people. Alpha level sims are perfect digital copies of a person developed though destructive scanning of a persons brain. Beta simulations are developed by observing every minute of a persons life and aggregating that data with an AI that learns and can perfectly predict a persons next thought or action. As it relates to GA or LLMs, if you had enough compute to have a GA shadow you for years, could it be possible that the LLM gets perfectly aligned to you?
Even writing all this out is bittersweet. It does seem somewhat obvious that these things are not impossible. Would a GA beta simulation perfectly replicate you? Maybe not, but it should would be a lot better than Claude. It strikes me that both government and corporations would fight all of this tooth and nail. Perhaps an open source path exists, but it seems really unlikely that it would ever get wide adoption. We will be funneled down the Claude path and it’s likely going to be total shit.
More options
Context Copy link
Of course I hate the idea of AI trained on the sensibilities of silicon valley elites or chinese communist commissars. But that's what's on the table right now. Grok is too shitty and Elon is too fickle to make his alternative worth using. And there's nobody around to put up enough money to make another good model.
In addition to his personalization shtick, Gwern also explores some other chatbot problems. I think each of these is potentially a research problem in its own right and can pay dividends unrelated to any personalization or alignment thing.
More options
Context Copy link
I don't like the "writing science fiction" characterization because it reads as inappropriately dismissive. We're not talking about holodecks or matter assemblers; we're talking about a technology that already exists. The only speculation involved is how much better it can actually get. You could in principle train and RLHF and RAG and whatever a personally-aligned model today, you probably just don't have the resources.
Gwern is 100% correct (and far from the first person to notice) that the current business model for AI is the same as every other tech business model in the information age: to take de facto and de jure control of as much of commerce and governance as possible and then rent it back to people in perpetuity. The open source software movement was a reaction to this decades ago. The main difference at present is that AI development is a lot more difficult to do at home (though of course not impossible!), especially while the current players have a hard lock on the anticipated production runs of all the necessary hardware. Memory and storage prices have quadrupled or more; for the first time in my life, desktop computers more than 3 years old have actually appreciated in value.
I agree with the "Guardian Angel" mission as stated. It's the only real effort I've seen from anyone to align AIs properly: with individual humans, rather than with corporate or government values. But I am not optimistic about their chances.
I am not sure a"Guardian Angel" is either possible or useful, but ignoring that... the worst trend among "rationalists" in the anti-AI movement is to propose that the average person just not be permitted to have some computing technology. Scott's Plan A is in this category, and on LW people have proposed that AI should not book a trip to a bullfight or should be made illegal for you to use if anyone abuses one of that model. Companies and governments don't like it that consumers have access to general purpose computing, and the "anti AI" movement is playing right into their hands.
Pressure from computer geeks has been the only thing that's helped at all against this trend, even to a weak degree. If the anti-AI movement makes computer geeks change sides, it'll be an apocalypse for general purpose computing even if the AI apocalypse isn't real.
More options
Context Copy link
More options
Context Copy link
What do I think? I think gwern is looking for a payday.
I wouldn't be surprised. In 2024 he claimed his salary was $12,000/year, in this interview
Although that may have changed, for example this guy claims he'd pay gwern $50-100k to live in SF, at least I'm sure the interview caused some increase to his Patreon.
Based on the most conservative estimate possible, he makes at least £2,000 per month from Patreon. It doesn't show the $ amount for me, but that's about $2,700. So $32,400 per year.
A more realistic amount would be double that (I can see £2, £4 and £8 tiers) so about $64,000 per year.
EDIT: It looks like you can join his Patreon without paying, so who knows?
If you scroll down on the About page you can see the number of paid members (280).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link