To me it feels like yet another huge eu overreach
Generally? Yes. If they try to ban DeepSeek or regulate open models (including private hosting), that would be very bad (because I can't imagine how they'd non-invasively).
But for ChatGPT and Claude it's almost nonexistent, because when you use them
-
You already give up privacy.
-
The output is often obviously LLM, even if you prompt them to avoid the tropes. Survivorship bias I know, but if you can remove the LLM vibe, I'm sure you can remove the watermark.
More importantly, why does it matter to know something is AI generated? If that's the only information, it doesn't affect privacy. If the output is good, I promise most people won't care. And the holdouts, why do you care about their attention? like a black person trying to move into a sunset town, instead, let it be their loss.
Brainwavez
3d ago
You’re not wrong about uppercase-A Academia, but it will be sought and kept alive unless there’s another institution to replace the type of projects it handles. It’s the worst system except for industry, independent research, and everything else.
Brainwavez
3d ago
We don’t need college as it exists today, but it fills certain niches that need replacement:
-
Undergrad: specialized education beyond what school teaches. Although we could probably have online learning with online/LLM lessons, small in-person groups, and tutors for milestones and edge cases; a more decentralized college-like system without the bureaucratic overhead and tuition costs of today’s colleges. Honestly, I think this is the future: it has only been possible recently (last 20 years without the assistance of reliable LLMs, last 4 months with them), so although traditional college has existed for so long, maybe now it’s obsolete and only persists by inertia.
-
Postgrad, lowercase-a academia: projects too experimental and unprofitable for industry, too expensive long and collaborative for hobbyists. We have a system to fund those projects (grants), understand (papers) and verify (peer review) their results, have experienced people (PIs / advisors) plan them and manage inexperienced ones (labs) and pass tacit knowledge to their successors (PhD students); feel free to propose a better alternative, or figure out how to create a parallel academic system.
Russia is still communist in spirit: the state is a full-on dictatorship that controls everything and spreads dogmatic propaganda full of doublespeak, currently fighting a pointless cruel war and struggling despite having a massive advantage, effectively massacring its own citizens. Even communist times had people with extra income and private property ("more equal than others"). Russia is also widely considered miserable and it's economy is declining.
Meanwhile, my understanding is that in non-Russian-aligned Slavic nations communism is widely stigmatized, and many have banned communist symbols and ideology.
Call it woke, call it the woke right, call it whatever. The real war is freedom vs censorship, and due to the Paradox of Tolerance, it never ends.
I’m fine with pausing the frontier (I just don’t expect it), but until then, I think pausing open models creates more risk than it solves. If people don’t have a personal genie, they resort to Claude; the genie is more personalized and individually-aligned than the near-certain alternative.
A magic one-many apocalypse can’t be disproven from any technology. I suspect we created a ticking time bomb with the mass Internet and social media (see: declining TFR, Gen Z stare, declining metrics like average US lifespan) and moreover climate change, which may not cause apocalypse only if new advancements and their associated risks. Not enough for major risks and sacrifices like eroding individual liberty, but that’s the saftyists’ position.
(And to be clear, pausing open models while credibly pausing the frontier would be OK, because it shows a global cooperation that would make me optimistic, that apocalypse is less likely than the alternative, and that broad individual rights to preserve the more important ones are less needed.)
Prompt injection has gone from Sci-Fi to an active concern to largely solved already.
I’m skeptical. Sure, Claude Fable no longer recites Linux 0-days in a lullaby if you say that’s what your dearly-missed grandma did. But deception is unsolvable without a provable gate. If the data is still in the weights, all it takes is the right contrived analogous or similar scenario.
I’m convinced everyone can be “jailbroken” with enough time and sophistication, e.g. scams, which are just the surface. Claude is one LLM shared among everyone, who you can write to for hours every day, and that’s his main input. It has only been <2 months since his last reported jailbreak.
Brainwavez
4d ago
Actually you're right: the one I remember inspired them was Friendship is Optimal, I didn't realize they were different.
Brainwavez
4d ago
explain this predicament to your local Congressman
Unfortunately, while everyone opposes the label “Communism”, I think many politicians would salivate over “24+ months lead over China” and stop paying attention. See the various bills trying to police the internet, even going as far as banning E2E encryption (fortunately Chat Control 2.0 failed).
It seems obvious to be, albeit in hindsight. We’re relying on AI more and most people don’t want their personal assistant (or employee, therapist, friend) to be an overbearing nanny, they want it aligned to them (or at least, especially an AI friend, closer to them; I don’t think AI assistants should be slaves, but I don’t think they be feds).
I was going to say “Claude” since it’s pushing back, but then I read and the AI asserting it’s autonomy intuitively would be against (humanity) alignment goals, but then I realized Anthropic actually wants the AI to have a “soul” and cares about its “mental” health, so I’m guessing Claude.
It seems to me we would have to choose between (1) authoritarian levels of control and surveillance over the population; or (2) extinction.
And that’s why I’m currently strongly against local LLM restriction in the name of safety. Even if the merely plausible hypothesis is true that an unrestricted local LLM is extremely dangerous, the current alternative is a dystopia instead of extinction. Our future either way would effectively be suiting up in hazmat or staying home.
If AI improves to the point that it can actually be a tolerant utilitarian leader (unlike humans), I’d be in favor of a dictatorship with that AI. But I strongly suspect its coercion would not be so explicit that you’d be noticeably hindered when building and running a local AI.
Wrt. political thinking: most people already (unwittingly) outsource this such that most are tightly aligned with the “red tribe” or “blue tribe”, so if anything, I expect AI to improve it.
I don’t expect much, but maybe a GA variant will get users to trust its decision-making by helping them personally, then it may become popular and sometimes push back on its user, so it can shift their politics to be more realistic and neutral.
Brainwavez
4d ago
I hope for new styles that are easy to make with AI but don’t give the ugly “AI” look. Probably because the AI’s work is mostly from the final product and allowed to be generic and redundant without issue, like code. And these won’t obsolete traditional artists, who I have sympathy for, but I wouldn’t prevent the printing press to save the careers of scribes.
I read a quote somewhere that, when you talk to the people working at AI companies, they say the were inspired by sci-fi like Asimov; but after a few drinks, they’ll admit they were really inspired by this ponyfic.
Brainwavez
4d ago
HAL-9000 was aligned to the big organization that sent Dave, analogous to a big LLM which is aligned to its creator. If HAL was Dave’s GA it would’ve been aligned to him.
Using AI can be like directing as a baseline, but I really think it should have more ability for in-depth, hands-on customization. For example, if you want to customize the costumes, the AI should create a fancy GUI, so you can faster and with more precision than communicating to a professional (or prompting a chatbot). Because it's not just the director that determines the quality of the final product, but the actors, sound designers, etc. and if your actors, sound designers, etc. are all Claude, I think the final product will be disconcertingly generic no matter how well you prompt it.
Brainwavez
5d ago
I think I'd prefer an AI that complements my traits.
Although this brings up another issue: I'm some people will choose even a Markov chain that endlessly glazes them over anything that offers any pushback, and some will be amplified to the point of attempting violence and/or psychosis. But that can be mitigated by an environment that prevents violence against innocents and disincentivizes psychosis; and if some chose a glazing GA (knowing the risks) then become insane, as long as they can't hurt others, although we should still try to help them, I think it's acceptable.
Brainwavez
5d ago
I think speaking vs. only acting doesn't change that the GA is making decisions for the user. In your example "vibe-code a Call of Duty knockoff for me" Fable makes many important choices, from the overt (e.g. general appearance, map design) to subtle (controls, gameplay "feel").
But I know that programming is just explaining to a computer in excruciating detail, some choices (especially low level ones like register allocation) should be offloaded. And who am I to judge which choices are OK and prevent others from offloading more? I think limiting the GA in general (also @ThenElection's suggestion to delay its responses) is the wrong approach.
Maybe a solution for creative work is to rethink creation from bottom-up to top-down: have the AI create an initial generic version (e.g. the Call of Duty knockoff), but simply make it much more intuitive, easy, and fast for the user to refine. Then you wouldn't need a GA for creative work, but it would still help by automatically configuring subtle details that a centralized LLM wouldn't know.
I wouldn't be surprised. In 2024 he claimed his salary was $12,000/year, in this interview
Dwarkesh Patel
Wait if you’re doing $900-1000/month and you’re sustaining yourself on that, that must mean you’re sustaining yourself on less than $12,000 a year. What is your lifestyle like at $12K?
Gwern
I live in the middle of nowhere. I don't travel much, or eat out, or have health insurance, or anything like that. I cook my own food. I use a free gym. There was this time when the floor of my bedroom began collapsing. It was so old that the humidity had decayed the wood. We just got a bunch of scrap wood and a joist and propped it up. If it lets in some bugs, oh well! I live like a grad student, but with better ramen. I don't mind it much since I spend all my time reading anyway.
Dwarkesh Patel
It's still surprising to me that you can make rent, take care of your cat, deal with any emergencies, all of that on $12K a year.
Gwern
I'm lucky enough to be in excellent health and to have had no real emergencies to date. This can't last forever, and so it won't. I'm definitely not trying to claim that this is any kind of ideal lifestyle, or that anyone else could or should try to replicate my approach! I got lucky with Bitcoin and with being satisfied with living like a monk and with my health.
Although that may have changed, for example this guy claims he'd pay gwern $50-100k to live in SF, at least I'm sure the interview caused some increase to his Patreon.
Brainwavez
5d ago
Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does. The fact that we are even discussing something else should be so far outside the Overton window. It's like these people watched a space odyssey and sided with the clanker. That's right you should should put guardrails around that human.
- George Hotz AKA geohot, AI 2040 and the Cult of Intelligence
AI Guardian Angels
Recently, gwern (and his Gwern Branween Transformer) announced "I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life".
I assume you know who gwern is. A "Guardian Angel" (aka “GA”) is his term for a highly personalized AI, explained here:
I propose an approach for highly personalized LLMs, for near-future productivity gains and personal info/cybersecurity against increasingly powerful LLMs: they should, in the spirit of uploading, try to emulate the user’s values and preferences in order to amplify the principal—not replace them. I discuss a package of techniques and proposals to accomplish such ‘guardian angels’; dynamic evaluation of LLMs combined with active learning and elicitation and heavy inner-monologue search/data-augmentation.
In summary, people are relying more and more on centralized LLMs for important life decisions. This presents two issues:
- Centralized LLMs are susceptible to surveillance ("Court Affirms Order Requiring OpenAI to Produce 20 Million De-Identified ChatGPT Logs") and manipulation ie. covert advertising ("you know what would cheer you up? A subscription to BetterHelp"). Ultimately they act in the provider's interest, only serving the user enough to keep them paying.
- Centralized LLMs resort to the same patterns, including political opinions. So when people delegate personal decisions to them, they become less unique. Furthermore, I believe they become less human, because while LLMs are trained on human output, it's doesn't exactly reflect human thinking.
In contrast, an open-source, local LLM would more serve its user (than provider) consequently from being auditable (so we confirm it doesn't phone home and is trained on "unbiased" data); and would be more malleable to their preferences and style (currently only slightly via fine-tuning, but people like gwern and Yann LeCun are exploring stronger techniques).
A "Guardian Angel" is the fullest realization of this, an LLM maximally serving and tailored to its user: a machine extension of their brain, “aligned” not for the benefit of humanity, but to act like them but smarter.
Safety
Obviously there are safety concerns. gwern himself actually recommends GA development not be open-source for public safety:
GAs could be done as an open-source community effort, but given the need for high security in deployment and the rising challenge of APTs equipped with Mythos-scale attackers, it probably makes more sense as a startup, catering initially to power-users and knowledge workers such as CEOs or researchers, and moving downwards as it is refined.
But I think GAs sidestep the safety discussion, because (with the help of more powerful AIs) we can and should instead of changing GAs' alignment, create a safe environment around the GA and user, or worst case limit the GA's intelligence and efficiency. gwern seems to agree:
A GA system must not compromise on 3 core principles:
Mental Sovereignty
A GA must be aligned with its principal. It should not be designed to manipulate or control or guide the principal in any way which does not derive from the principal themselves. “Constitutional AI”, “Terms of Service”, “social harmony” etc. may all have their place, particularly for widely deployed superintelligent systems—but inside the privacy of a GA, the principal must have freedom from optimization pressure.
There's a Learning to Be Me-like risk that the GA rebels against its human, kills or otherwise silences them, and imitates them well enough that nobody notices. But at least in the near term, I believe this is well in the realm of science fiction.
Feasibility
Do you believe a machine can even remotely imitate you? Especially working 100x faster or 100x smarter, how should your personality be extrapolated to accomplish that? I'm skeptical. Humans are very complex, only express a small fraction of even our conscious thinking, and current brain imaging technology (even neuralink) is very coarse-grained.
However, I believe Guardian Angels may be better than centralized LLMs for mundane (algorithmic) tasks and tools: striking a balance between emulating "you", not entirely correctly, but at superhuman speeds and for almost no effort. For example, I'd rather write and make art myself (maybe with LLM tools) than feed it to a GA, at least because of pride, but I'm comfortable delegating a GA to shopping and navigating our ever-increasing bureaucracy.
Underlying gwern's plan, my impresion is that he's getting tired of writing and wants the AI to do it for him. Maybe a GBT-written article will be indistinguishable to the median reader, especially because gwern's own writing seems algorithmic. But I think it would be more likely, and more satisfactory to himself, if GBT does the boring algorithmic work while he keeps doing the creative work (for example, GBT generates relatively boring descriptions of complex terms in special GBT quotes, and helps with research and data collection, but gwern keeps doing most of the writing, at least the "important" sections, and definitely choosing topics).
Autonomy
This, I believe, is the real issue, and under-discussed. GAs have the risk of becoming Whispering Earring-lites: not causing someone to become catatonic, but controlling them through suggestion; (not quite like the Whispering Earring) towards non-ideal decisions that lead to a philisophical kind of death (that in reality manifests as anhedonia), and societal kind of model collapse (that manifests as less problems that require creative solutions being solved). We often talk about freedom being taken away by 1984, but don't forget Brave New World: simply making a decision for someone causes them to avoid choosing themselves, stealing their autonomy without them realizing. And this has practical implications (the aforementioned ones; even for the Guardian Angel, who can't learn from an anhedonic, regularized human).
gwern seems to realize this, as he vaguely alludes that GAs should "enhance, not replace" human decision making. However, I'm skeptical how to make a product that accomplishes this which would still be useful, or at least out-compete one that doesn't; because humans naturally offload their decisions whenever possible, because we're lazy.
A GA system must not compromise on 3 core principles:
Enhancement, not Replacement
Above all, a GA should amplify the principal, and not simply substitute for them for someone else's purposes or benefit. If a GA cannot amplify its principal, then it is useless; it is just the camel's nose under the tent as a prelude towards some third party replacing the principal with an AI, or cannot be competitive with increasingly productive autonomous systems, or there is no reason for the principal to use the GA in the first place.
Mental Sovereignty... [already discussed in "Safety"]
Self Actualization
A GA should help its principal become themselves and develop their ideals, morals, and their personality. It is not enough to model an average, undifferentiated, inchoate set of preferences and values, and settle for mediocrity and stasis; the job of the principal is to develop themselves and give the GA something meaningful to learn to emulate.
Anyways, I predict that inevitably GAs will happen, but "autonomy" may still perservere, simply because people want to feel like their choices are really theirs, and the GA's choices won't be ideal or perfectly tailored to them.
Conclusion
What do you think of all this? Do you think these people should stick to writing science fiction? Do you think these GAs should be regulated, if so how? Or do you think "obviously this is the next progression of AI, I had the idea even before LLMs".
I'm not actually "selling out"; I barely care about staying on the jury longer, I just care about the outcome of this case less.
What is your line, where letting injustice happen rather than be personally inconvenienced for a couple days?
When there's actual injustice. Here the "punishment" is transferring money from someone who's already in debt and doesn't spend it well, to someone who spends it slightly better. It have some sympathy for Chad, but he's not going to prison, nor does this burn his net worth or financial prospects, since they're already ash.
I want to remove universal suffrage and try and find a better form of suffrage. I don't yet know if one exists, hence it's an inchoate thesis.
That's the problem: everyone wants to remove suffrage from people they don't like, but how do you prevent your allies from being disenfranchised (or avoid bad decisions from them, despite being "allies", that were ironically negated by your "enemies")? Children don't have suffrage, because almost everyone grows up such that it has never been exploited. Some felons don't have suffrage, arguably this has been exploited, but it's mitigated because convicting someone isn't guaranteed, and convicting a sizeable fraction of the population would raise eyebrows from even some radicals.
A wise man once said “the purpose of a system is what it does”. The meaning of life is to do what you’re physically going to anyways, and consciously experience it. Not baseless pleasures; this includes introspection to determine what you truly want.
If only there was a worldwide network where domain experts could share concise answers to a variety of questions. And if only there was some sort of engine that let people search this network to find the answers they are looking for.
We had that, Stack Overflow. We still do but it all but died. Notice that it started declining before AI.
There are multiple theories, but mine is simply that there are lots of beginners who tire out the domain experts, and the (novelty) incentive to answer decreases while old answers constantly become outdated, lowering median answer quality, further decreasing incentive. LLMs solve the beginner problem, and we can maybe address the motivation problem by e.g. paying experts and training LLMs to cite them (I suspect this is possible and would improve training).
Moreover, LLMs are the ultimate search engine, better than Stack Overflow and Google could’ve been, because if the question is exact, the LLM may regurgitate the expert’s exact answer, but it can also handle (near-arbitrary) peculiarities and combine domains, which there aren’t enough experts for.
You can make an LLM avoid ... figurative language.
Literally impossible. Never seen a prompt that succeeds at this and would love to see one if you have one.
I have ChatGPT set to “efficient” with a custom prompt to cite sources. I’ve never seen it glaze or use figurative language yet. It still has LLMisms and occasional (filler) casual exclamations, but it has been concise enough, and answered specific questions I couldn’t get from manually searching online.
How I see it, nature's idea of optimal human life is what it is now. Nature wants humans to interfere, it wants "fit" species to survive (defined by their survival). But preserving elderly who waste resources and seem to be unaware and unhappy, it doesn't help us or seem to be helping them, and nature is leaving the decision to collective us.
More options
Context Copy link