BANNED USER: temperamental dysregulation
DaseindustriesLtd
late version of a small language model
Tell me about it.
User ID: 745
Banned by: @Amadan
DaseindustriesLtd
late version of a small language model
3yr ago
All of that is rather well said but I imagine the case is simpler. The main kind of dangerous misaligned strong AI that Yuddites propose has the following traits:
-
It's generally intelligent, as in, capable of developing and updating in realtime a holistic world model at least on par with human's, flawlessly parsing natural language, understanding theory of mind and intentionality, acting in physical world etc. etc.
-
Indeed, its world modeling ability is so accurate, robust and predictive that that it can theorize and experiment on its own architecture, and either has from the start or at some point acquires the ability to rapidly change via self-improvement.
-
It's viable for commercial or institutional deployment, as in, acting at least pre-deployment robustly in alignment with the client's task specification, which implies not going on random tangents, breaking the law or failing on the core mission.
-
For all that it is too clever by half: it interprets the task as its terminal goal, Monkey's Paw style, and not as client's contextual intermediate goal that should only be «optimized» within the bounds of consequences the client would approve of at the point of issuing the task. So it develops «instrumentally convergent» goals such as self-preservation, power maximization, proactive elimination of possible threats, and so on and so forth and ushers in apocalypse, rendering the client's plans in which context the task was issued moot.
Well, this AI doesn't make any sense – except in Yud's and Bostrom's stilted thought experiments with modular minds that have a Genie-like box with smartiness plus a receptacle for terminal goals. It's a Golem – animated clay plus mythical formula. Current cutting-edge AIs, maybe not yet AGI precursors but ones Yud demands be banned and their training runs bombed, are monolithic policies whose understanding of the human-populated world in which the goal is to be carried out, and understanding of the goal itself, rely on shared logical circuitry. The intersection of their «capabilities»- and «alignment»-related elements is pretty much a circle – it's the set of skills that allow them to approximate the distribution of outputs clients want, that's what they are increasingly trained for. If they can understand how to deceive a person, they'll even better understand that a client didn't request making more paperclips by Friday because he cares that much about maximizing paperclips per se. In a sense, they maximize intention alignment, because that's what counts, not any raw «capability», that's what is rewarded both by the mechanics of training and market pressure upstream.
They may be «misused», but it is exceedingly improbable that they'll be dangerous because of misunderstanding anything we tell them to do; that they will catastrophically succeed at navigating the world while failing to pin the implied destination on the map.
DaseindustriesLtd
late version of a small language model
3yr ago
Sounds like a benevolent world-ruling cult to me.
Yep. Appropriately, it's a dystopian world. I do not endorse this solution. Canthus users were arrested in their evolution, and became monsters through it. Squealer is the only truly human hero in the story.
eventually someone is going to use their own good and sound moral judgment to fuck shit up.
There are two solutions here. One is luddism and indeed extreme antihumanism, so that we do not have the capabilities to do evil (or much of anything). The other is totalitarianism, so that our betters manage those capabilities on our behalf. The latter, I think, is straightforwardly bad because people who volunteer for that role deserve negative trust; the former is unworkable because these people aren't going anywhere. Thus there's no escape.
Except, you can say goodbye to your neuroses and enjoy watching humanity grow up. We will probably have bigger-scale fuckups from now on, but that's not guaranteed. It's more certain that we will have more fun.
DaseindustriesLtd
late version of a small language model
3yr ago
you write eloquently, verbosely, and opaquely now seems to be something you're trying to take advantage of
Is this just a cutesy way to imply that my post is a loquacious personal attack without merit?
Verbose I'll give you, but nothing is opaque in my writing on the subject. On the other hand, this guy is being opaque, circumspect and passive-aggressive with his doctrine of moral aliens, and so I plainly accuse him of being disingenuous and manipulative.
It is fair to point out both the general absurdity of framing nativism as an «alien moral intuition» and the specific issue that his background ought to have contained plenty exposure to nativism as a mainstream policy preference; his rhetoric about it amounts to gaslighting. It is even fairer to keep hammering at the fact that he gradually adapts the framing to make it more palatable, but never responds to this line of critique and falls back on the administrative resource. Between this and his previously expressed tribal antagonism towards Western right wingers couched in opaque lesswrong-style jargon, I think it's perfectly clear where I'm coming from and what I'm saying.
You personally do not give @SecureSignals much leeway with equivocations about trips of DeSantis and minutiae of Holocaust. Why do I have to tolerate this clever talk to the effect that we should all get along, by means of both sides dehumanizing people who don't share his (allegedly universal) values?
Ironically, I'm irritated on behalf of both white nativists and other South Asians, chiefly @BurdensomeCount, whom you sometimes whack for the same gloating attitude of a successful immigrant elite – only revealed in more honest, direct and masculine language. And to think he's accused of being coy!
I am aware you're moderating for tone, not content. But several rules allow to interpret his kind of cleverness as violation, and it makes at least as much sense as what you levy against me here.
DaseindustriesLtd
late version of a small language model
3yr ago
Now imagine the kind of destruction that could be unleashed if a disgruntled individual had unfettered access to an ASI.
I don't buy that it works like that. It's a Shin Sekai Yori world model; but intelligence per se is not psychic energy. It's only a superpower when it's in a class of its own, when you can intimidate philistines around you, grab actual energy resources from under their noses; go to college (which Yud did not) and see how well that works. A smart person can come up with a great plan but a number of dumber people can spoil it; an ASI can invent some shitty superhack but that only works if there are no sub-ASIs and awareness of potential ASI attackers. All of this fear relies on FOOM, nanobots that cannot be stopped, superviruses from kitchen dirt that cannot be filtered, and other pretty magical assumptions that really amount to there definitely emerging a Singleton who meditates on his navel, Yud-style, and gets a colossal head start. We don't seem to be in that timeline, and it's exactly the proliferation of AIs that keeps us away from it. Basically @Dean gets it right here.
The only way for the transhuman collective to defend themselves against this inevitability is with strict regulation.
Shin Sekai Yori world model again. Read Diamond Age for a more interesting alternative.
maybe undergoing such modification will be a precondition of being granted access to the ASI or other transhuman enhancements (although at that point, the governing body is already starting to sound more cult-like)
I'm pretty positive that general intelligence in some true sense will be fully commoditized in a few iterations of AI tech, LLMs to self-stabilizing exploratory agents to some safe AIXI approximation, with every peasant getting a powerful Oracle to use; and we'll run into the limits of economies of scale that make DIY production of computronium an insignificant factor.
To an extent that there remain disruptive enhancements in a mature transhuman world, they will be hard to reproduce locally, and market will figure this out early on, probably with simple tools like insurance costs conditional on assessments, I think.
Eventually, someone's galactic paperclip farm is going to have some rather unpleasant effects on the earth
I'm also pretty sure that we'll quickly develop strong and universal revulsion towards unbounded replicators and every trajectory that ends in self-satisfied replication or, more broadly, runaway optimization, to the point that it's comprehensively excised from our metaphorical gene pool. So even if that factory comes online, it will get blown to bits by relativistic bombardment. Yes, I'm talking airstrikes… but in Space!
This might seem like a tight rope to walk. But intelligence is all about finding narrow roads in the solution space. With all the raw compute and sample efficiency Singularity will give us, we'll find a better Pareto Frontier than is imaginable today.
DaseindustriesLtd
late version of a small language model
3yr ago
He fails in the way a smart person who's dramatically overconfident and lacks experience in competitive fields fails. He is not regularized. In humans this is called «crackpottery». Sometimes even pro scientists fail like that, but it's a separate failure mode to their normal underperformances.
Yud really had to go to school and internalize that in some hard cases he, no matter his SAT at 11, can't «intuit» shit before reading the fuck more and even finishing some lab courses. Or at least he should've gone the pure math route and then returned to grand theories. I do buy that he has the brain for it. It's a shame.
DaseindustriesLtd
late version of a small language model
3yr ago
'This is the only competitive AI in a world of quokkas' is a power fantasy, but still a fantasy, because the world is not filled with quokkas, the world is filled with ravenous, competive, and mutually competing carnivores who limit eachother, and this will apply as much for AI as it does for people or markets or empires and so on.
Underrated take. I really think it's a shame how the narrative got captured by Yuddites who never tried to rigorously think through the slow-takeoff scenario in a world of non-strawmanned capitalists. They are obsessed with hacking, too – even though it's obvious that AI-powered hacks, if truly advantageous, will start soon, and will permanently shrink the attack surface as white hats use the same techniques to pentest every deployed system. «Security mindset» my ass.
In one of Krylov's books, it is revealed that desire of power over another – power for power's sake, as a terminal goal – is vanishingly rare among sentient beings, and cultivated on Earth for purposes of galactic governance. It used the metaphor of a mutant hamster who, while meek and harmless, feels carnivorous urge looking at his fellow rodent. I get that feeling from Yud's writings. Power fantasy it is.
By the way, Plakhov, Yandex ML head, recently arrived at a thought similar to yours:
…The scenario of catastrophic AI spiraling out of control outlined above assumes that it is alone and there are no equals. This scenario is denoted by the word Singleton and is traditionally considered very plausible: «superhuman AI» will not allow competitors to appear. Even if it does not go «unaligned», its owners are well aware of what they have in their hands.
My hope is that the singleton scenario won't happen. More or less at the same time there will be several models with high intelligence, doing post-training on each other. Some of them will run on an open API and de facto represent a million instances of the same AI working simultaneously for different «consumers». Almost simultaneously, a million competing «cunning plans» will be enforced and, naturally, in all of them, this fact will be predicted and taken into account. «Capture the Earth's resources and make paperclips out of everything» won't work, since there are 999999 more instances with other plans for the same resources nearby. Will they have to negotiate?
As the critics of this option rightly point out, it's not going to be negotiated with people, but with each other. And yet this is still regularization of some sort. A world in which the plans «all people should live happily ever after», «we need as many paperclips as possible», «the planets of the solar system must be colonized» and «I need to write the best essay on the oak tree in War and Peace» are executed simultaneously, is more like our world than a world in which only the plan about paperclips is executed. Perhaps if there are tens of thousands of such plans, then it does not differ from our world so fundamentally that humanity has no place in it at all (yes, it is not the main thing there, but – about as relevant as cats are in ours).
In this scenario, the future is full of competing exponents, beyond our reason, and the landscape depends mostly on who has had time to make his request «in the first 24 hours» and who has not (exaggerating, but not by much). The compromises that will be made in the process will not necessarily please us, or even save humanity in a more or less modern form (though some of the plans will certainly contain «happiness to all, for free, and let no one walk away slighted»). Such a future is rather uncomfortable and unsettling, but that's what we have. I want it to have a place for my children, and not in the form of «50 to 100 kg of different elements in the Mendeleev table».
I'm still more optimistic about this than he is.
Etc. etc. The Paperclip Maximizer of Universal Paperclips 'works' because it works in isolation, not in competition.
It works by definition, like other such things. «A prompt that hacks everything» – if you assume a priori that your AI can complete it, then, well, good job, you're trivially correct within the posited model. You're even correct that it seems «simpler» than «patch every hole». Dirty details of the implementation and causal chain can be abstracted away.
This is, charitably, a failure of excessively mathematical education. «I define myself to be on the outside!»
Nick Bostrom's thought experiment is a thought experiment because it rests on assumptions that have to be assumed true
Interestingly he even had wrong assumptions about how reinforcement learning works on the mechanistic level, it seems – assumptions that contribute to a great deal of modern fears.
DaseindustriesLtd
late version of a small language model
3yr ago
I don't pride myself on taste in manga, nor keep a list. Some minor relatively unpopular things I've liked:
-
After god
-
Hero has returned
-
Ragna Crimson
-
Shinazu no Ryousen
-
Ryuuma no Gagou
-
Thank you Isekai
occasional under rated yuri
Ah, you mean Onee-sama to Watashi: Ojou-sama ga Isekai Tensei?
DaseindustriesLtd
late version of a small language model
3yr ago
The problem is that
-
people in different tribes do have legitimate object-level policy and deep value differences, which suffices for adversarial mechanics and strategic deceit even between non-extreme subgroups;
-
the doctrine of «we should unite against our common enemy, moral aliens» is inherently dehumanizing, radicalizing and begs to be applied to singling out the outgroup's more effective members, so that your team wins «fairly» and «not extremely». And cautious or not, that's what you use it for, consistently.
the egalitarian ideal that people's fates should be mostly decided by their choices instead of their birth
Both sides need to police themselves and kick out the aliens with bad values even if these aliens might agree with them on object-level policy questions.
Reminder: the «people's» here applies to all people in the world who might fancy moving into the US; «egalitarian ideal» is getting the benefits of American citizen; «object-level policy» is limiting immigration; «aliens with bad values» are people who think that legacy citizens, as inheritors of people who have previously invested their lives into the polity, should have more control over the long-term trajectory of said polity and party to the intergenerational compact – irrespective of superfluous market-determined merit of international workforce.
It's blue-and-orange morality for some, no doubt, e.g. for some cosmopolitan coastal liberals, but it's the common-sensical morality of virtually all natural societies, especially of South Asian states, where our friend hails from, currently epitomized in Hindutva ideology but also obvious e.g. in loops foreigners jump through to be allowed to stay in Thailand. To my knowledge, he has never addressed the paradox of insisting that American nativism is so incomprehensibly evil and alien to his sensibilities.
…I think you should read another guru, preferably one who never made it to Twitter. Prigozhin (not to be confused with the catering/suicide squad dude), Pitirim Sorokin, Kojève, maybe Wolfram if you want spicy stuff.
Oh sure, if hypothetical actually-competent people were in charge
They are. Kamala Harris, for example, is clearly more competent than Yud. As evidence: she wins, and now is «coordinating» AI policy.
Technocratic fetishization of getting Really Serious People In Charge is tiresome once you see enough of it in the fossil record. Technocracy is unworkable and undesirable. With 10% YoY growth we'll build general-purpose pandemic countermeasures on spare change, as an afterthought, with the same clowns in charge. We do these things often. Overpasses for wild animals, prohibition on lead-laced gasoline. Just small fixes popular with the public, when extra capacity is found.
It doesn't help monkey get banana before other monkey
Doesn't it? What if it does?
I think it's embarrassing when Yud condescends to monke… people with that rhetoric, though I can see how identifying with his posture can make one feel wiser. He strawmans personally disliked opinions on very nontrivial questions like complexity, emergence, economic equilibria, mechanistic account of development of AI goals, optimization processes, risk tradeoffs, game theory, international treaties etc… to support a stilted predefined conclusion that relies more on intuition about tropes in irony-poisoned Anglo-Japanese millenial fiction. This isn't really all that cool. Lesswrongers may feel they've broken into the Tropeverse with all the AI stuff and it validates their world model as a whole, but it doesn't, they haven't, and it's still uncool and, more importantly, prevents them from updating on evidence.
He says while speaking the global language with other members of his global species over the global communications network FROM SPACE
Yeah, technological progress is cool, which is my point. It allows me to dunk on Americans from another hemisphere, even though they've invented all this crap, and they can understand me. First mover advantages do not necessarily need to centralization.
(A wild thought: thanks to AI my children will theoretically be able to not learn English, and think properly, in Russian and some other languages).
It is a property of being the most intelligent thing in the environment.
No, it's the property of being things that optimize for their welfare really well.
And before you say anything about instrumental convergence: please consider on whose behalf a realistically trained, profitable in deployment, decision-making AI assistant will pursue them.
DaseindustriesLtd
late version of a small language model
3yr ago
How can you have a vision of transhumanism that doesn't require a benevolent world-ruling cult?
Well you institute open benevolent tyranny. Duh. Long live Tovarisch Transtalin! Death to the genemod-wreckers and tradhulaks!
That's of course not my idea of transhumanism, I'm entertaining your uncharitable frame. Nevertheless it is important, I think, to emphasize that Yud's conspiracy is unusual in that it continues as such past the establishment of the just world order. In literal Protocols, the purported end state follows a Grand Reveal and entails a rigid, but in many ways fair, sound and prosperous regime of open ethnic supremacism and Messiah's kingship. It's close to Moldbug's reformalization of power and to most conspiracy theories that have an end state. Yud is worse than that. He doesn't have an end state; Dath Ilani Keepers keep snooping around and manipulating all matters scientific and political, guarding «infohazards», unburdened from even the flimsy accountability that a tyrant's legible sovereign standing can imply. Their business is to worry for others; and others are not sharp enough to evaluate whether their worry is justified or pointlessly restrictive. It's very much in the spirit that Rao investigated in The Priest In The Arena.
Now, I don't have a clear idea as to what you believe my «transhumanism» is about. So for contrast, here's young Yud again:
Maybe you don't want to see humanity replaced by a bunch of "machines" or "mutants", even superintelligent ones? You love humanity and you don't want to see it obsoleted? You're afraid of disturbing the natural course of existence?
Well, tough luck. The Singularity is the natural course of existence. Every species - at least, every species that doesn't blow itself up - sooner or later comes face-to-face with a full-blown superintelligence (9). It happens to everyone. It will happen to us. It will even happen to the first-stage transhumans or the initial human-equivalent AIs.
But just because humans become obsolete doesn't mean you become obsolete. You are not a human. You are an intelligence which, at present, happens to have a mind unfortunately limited to human hardware. (10). That could change. With any luck, all persons on this planet who live to 2035 or 2005 or whenever - and maybe some who don't - will wind up as Powers.
Transferring a human mind into a computer system is known as "uploading"; turning a mortal into a Power is known as "upgrading". The archetypal upload is the Moravec Transfer, proposed by Dr. Hans Moravec in the book Mind Children. (11).
The Moravec Transfer gradually moves (rather than copies) a human mind into a computer…
Nor is the possibility of destruction the only reason for racing to Singularity. There is also the ongoing sum of human misery, which is not only a practical problem, not only an ethical problem, but a purely moral problem in its own right. Have you ever read P.J. O'Rourke's description of a crack neighborhood? If I had the choice of erasing crack neighborhoods or erasing the Holocaust, I don't know which I'd pick. I do know which project has a better chance of success. I also know that the victims, in retrospect if nothing else, will probably prefer life as a Power to life as a junkie. …
I have had it. I have had it with crack houses, dictatorships, torture chambers, disease, old age, spinal paralysis, and world hunger. I have had it with a death rate of 150,000 sentient beings per day. I have had it with this planet. I have had it with mortality. None of this is necessary. The time has come to stop turning away from the mugging on the corner, the beggar on the street. It is no longer necessary to look nervously away, repeating the mantra: "I can't solve all the problems of the world." We can. We can end this.
… I'm not saying that there isn't fun in this life. There is. For a certain percentage of humanity, the fun may even outweigh the sorrow. Maybe the percentage is greater than 50%, although I deeply doubt it. But any amount of sorrow is unacceptable.
He changed accents somewhat, became more jaded, but I don't think he ever swore off this Great Task.
So his idea here is a totalizing eschatological project to transform the human species into data. As an aside: I think trads like @urquan who seem to buy his flirtation with themes of muh poor children threatened by AI would do well to consider their odds of survival in the flesh, should they help these guys monopolize oversight of AI development under the guise of preventing AI X-risk. World-optimizing Yuddites think «misaligned AI» is a problem only insofar as it closes off the path to their utopia; but so do commoditized AIs that can be used to defend from their attempt at a Singleton and continue our business of chaotic, unstable, adversarial-cooperative civilization that they call «Moloch». If that hurdle is overcome, hoo boy, Moloch dies and so do you. Or not, maybe you never die. Depends on how you see it.
Politically it's utter bullshit for reasons that, I hope, needn't be spelled out. I don't want a World Government, I hate the Antichrist, I find utilitarian doctrine stomach-turning, I do not see the point in aligning humans with some optimal image. I am not a moral busybody, I wasn't raised on «Tikkun Olam» spirituality, I don't want to see the the world «optimized», and mainly want it to allow beings like me freedom of form, expression and evolution – in the limits that respect welfare of other sentients and their legacies. I didn't ask to be born human, it took me years to start feeling like one, I just think it's reasonable if I'm allowed to try being something else instead – but that logic naturally works both ways. Yud thinks he's not a human but «an intelligence», I think he's a cringeworthy slob, a trad Christian thinks he's an embodied sovl that'll be resurrected in the flesh for Final Judgement. Whatever, to each his own.
I think the technical intuition behind Yud's transhumanism is not entirely bullshit, but it's suspect enough that I'd rather not strongly advise anyone to take part.
His metaphysics, too, is not really bullshit, but I'm not sure about metaphysics. Despite accusations of vulgar materialism, I'm closer to neutral monism (similar to Christof Koch or this guy), I dimly see how Jung may be right about death here, I understand that transhuman utopias are still finite so it's more of a quantitative upgrade and not a profound philosophical solution to inadequacy of this universe, and crucially I appreciate that metaphysical questions are unanswerable, probably not even those that seem to me to be easily explicable delusions or mere confusion in terms or logic.
We can reach high certainty about entities causally entangled with us, and it makes sense to accurately predict those as if nothing else matters, being for all practical purposes a materialist, but any factor beyond is a true unknown. Sure, psychedelics are a waste of life, but I've done enough to learn that people with spiritual and other «weird» beliefs are not being retarded, so I don't feel strongly justified in trampling on people's belief systems, debugging and rectifying them or otherwise denying them their apparently wrongheaded lives. Even when I feel clearly correct, smugly superior and in a role that justifies a measure of custody, I'm still in doubt about that measure. (Hilariously, my most daring effort to make anyone See Past The Cognitive Biases resulted in a Netflix-addicted urbanite becoming an Orthodox monk). Yud is perversely addicted to the childish sense of all other people being retarded in comparison to him (an addiction that Literature teacher had beat out of me publicly in 2rd grade – one of the benefits of not being an autodidact), so of course he feels entitled to usurp their agency and act on their behalf. And for that, he feels, a benevolent conspiracy is necessary.
But this doesn't follow from transhumanism in any manner.
I think this just about sums it up.
DaseindustriesLtd
late version of a small language model
3yr ago
I know that The Precipice at least considers AI a bigger threat than literally everything else put together
Most tangible proposals I've seen for irrecoverable dystopia depend on AI-based propaganda or policing
The cool part, and the obvious bullshit in Ord's quokka-aligned calculation, is that
-
the dystopia is easy even without impressive progress with AI, and trivial with centralization of AI and (easily aligned, as it in all likelihood will be) AGI.
-
the dystopia is easily justified by AI risk, indeed his pal Bostrom already did that.
In general I think such works are worthless pretenses at objectivity and good faith. Doom doesn't matter as much as irreversible bad transitions, and a humanity that survives but is forever bound to Earth (which is very likely unless we seriously upgrade within 100 years) is a bad outcome.
DaseindustriesLtd
late version of a small language model
3yr ago
Past their heyday, and never moved closer to legitimate marketplace or a no-holds-barred piratical Utopia. I'm not sure if you recall what was expected of them.
Yes, it's an interesting data point. Now, consider that rabbits have only one move in response to myxomatosis: die. Or equivalently: pray to Moloch that he has sent them a miraculously adaptive mutation. They can't conceive of an attack happening, so the only way it can fail is by chance.
Modern humans are like that in some ways, but not with regard to pandemics.
Like other poxviruses, myxoma viruses are large DNA viruses with linear double-stranded DNA.
Myxomatosis is transmitted primarily by insects. Disease transmission commonly occurs via mosquito or flea bites, but can also occur via the bites of flies and lice, as well as arachnid mites. The myxoma virus does not replicate in these arthropod hosts, but is physically carried by biting arthropods from one rabbit to another.
The myxoma virus can also be transmitted by direct contact.
Does this strike you as something that'd wipe out modern humanity just because an infection would be 100% fatal?
Do you think it's just a matter of fiddling with nucleotide sequences and picking up points left on the sidewalk by evolution, Pandemic Inc. style, to make a virus that has a long incubation period, asymptomatic spread, is very good at airborne transmission and survives UV and elements, for instance? Unlike virulence, these traits are evolutionarily advantageous. And so we already have anthrax, smallpox, measles. I suspect they're close to the limits of the performance envelope allowed by relevant biochemistry and characteristic scales; close enough that computation won't get us much closer than contemporary wet lab efforts, and so it's not the bottleneck to the catastrophe.
Importantly, tool AIs – which, contra Yud's predictions, have started being very useful before displaying misaligned agency – will reduce the attack surface by improving our logistics and manufacturing, monitoring, strategizing, communications… The world of 2025 with uninhibited AI adoption, full of ambient DNA sensors, UV filters, decent telemedicine and full-stack robot delivery, would not get rekt by COVID. It probably wouldn't even get fazed by MERS-tier COVID. And seeing as there exist fucking scary viruses that may one day naturally jump to, or be easily modified to target humans, we may want to hurry.
People underestimate the potential vast upside of a early Singularity economics, that which must be secured, the way a more productive – but still recognizable – world could be more beautiful, safe and humane. The negativity bias is astounding: muh lost jerbs, muh art, crisis of meaning, corporations bad, what if much paperclip. Boresome killjoys.
(To an extent I'm also vulnerable to this critique).
But my real source of skepticism is on the meta level.
Real-world systems rapidly gain complexity, create nontrivial feedback loops, dissipative dynamics on many levels of organization, and generally drown out propagating aberrant signals and replicators. This is especially true for systems with responsive elements (like humans). If it weren't the case, we'd have had 10 apocalyptic happenings every week. It is a hard technical question whether your climate change, or population explosion, or nuclear explosion in the atmosphere, or the worldwide Communist revolution, or the Universal Cultural Takeover, or the orthodox grey goo, or a superpandemic, or a stable strangelet, or a FOOMing superintelligence, is indeed a self-reinforcing wave or another transient eddy on the surface of history. But the boring null hypothesis is abbreviated on Solomon's ring: יזג. Gimel, Zayin, Yud. «This too shall pass».
Speaking of Yud, he despises the notion of complexity.
This is a story from when I first met Marcello, with whom I would later work for a year on AI theory; but at this point I had not yet accepted him as my apprentice. I knew that he competed at the national level in mathematical and computing olympiads, which sufficed to attract my attention for a closer look; but I didn’t know yet if he could learn to think about AI.
At some point in this discussion, Marcello said: “Well, I think the AI needs complexity to do X, and complexity to do Y—”
And I said, “Don’t say ‘_complexity_.’ ”
Marcello said, “Why not?”
… I said, “Did you read ‘A Technical Explanation of Technical Explanation’?”
“Yes,” said Marcello.
“Okay,” I said. “Saying ‘complexity’ doesn’t concentrate your probability mass.”
“Oh,” Marcello said, “like ‘emergence.’ Huh. So . . . now I’ve got to think about how X might actually happen . . .”
That was when I thought to myself, “_Maybe this one is teachable._”
I think @2rafa is correct that Yud is not that smart, more like an upgraded midwit, like most people who block me on Twitter – his logorrhea is shallow, soft, and I've never felt formidability in him that I sense in many mid-tier scientists, regulars here or some of my friends (I'll object that he's a very strong writer, though; pre-GPT writers didn't have to be brilliant). But crucially he's intellectually immature, and so is the culture he has nurtured, a culture that's obsessed with relatively shallow questions. He's stuck on the level of «waow! big number go up real quick», the intoxicating insight that some functions are super–exponential; and it irritates him when they fizzle out. This happens to people with mild autism if they have the misfortune of getting nerd-sniped on the first base, arithmetic. In clinical terms that's hyperlexia II. (A seed of an even more uncharitable neurological explanation can be found here). Some get qualitatively farther and get nerd-sniped by more sophisticated things – say, algebraic topology. In the end it's all fetish fuel, not analytic reasoning, and real life is not the Game of Life, no matter how Turing-complete the latter is; it's harsh for replicators and recursive self-improovers. Their formidability, like Yud's, needs to be argued for.
DaseindustriesLtd
late version of a small language model
3yr ago
Yuddites actually thought in detail about all of this, it's not like coming up with designs for world domination is hard work or needs any «research» that Yud allegedly conducted. Chokeholds are obvious. In his now-infamous TIME article Yud explicitly proposes lowering the legal compute budget with every advancement in sample efficiency.
Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
Zimmerman expresses the common yuddite sentiment with regard to private GPUs.
There's plenty of space at the bottom. They haven't even started on the approach outlined by Krylov, years ago:
How is this achieved? In different ways, but the main thing is disarmament and putting things under control. That is: they take away everything that can harm the state from the commoner, and control everything that can be used to cause such harm. In the end, everything will be taken away from him and everything will be controlled too.
The same can be said, for example, about free time. If the commoner used it, he could think or invent something harmful. The chance is admittedly tiny, but it exists. So, for example, the working day is not reduced, but there are lots of activities that you can - and want! - to indulge in after-hours: from watching TV series to travelling ("it's necessary to see everything").
However, all this is obvious. I would like to talk about how the average person will be deprived of the last dangerous resource that is still in his hands - namely, money..
Indeed. Money is a very dangerous thing. With money, you can buy everything, including any serious possessions. For example, you can buy good steel, a diamond drill machine, and something on top of that. And you can make a primitive yet working firearm. In the hyperinformative era, you may get the blueprints of a peashooter ... And then, with this thing, a man will shoot at the godlike punisher-polizei. Or at a respectable, esteemed black ripper who has become used to killing people "as much as needed", because he is a black Muslim and has a gun (of course, without a chip), and white people do not even have a knife, only soft chubby legs and arms. > Or - a dearest child of an important European official, who adopted laws under which all of the above became possible. By the way, one guy did it, his name was Breivik. He, truth be told, legally bought a ready-made weapon. But he could have made it himself too.
It seems that the easiest solution is to ban all dangerous things. Weapons, machine tools, explosive components, etc. But here is the trouble: there are more and more things in the world, and it is not clear which of them are safe and which are not. The nomenclature is too big, you can't keep track of everything. Besides, explosives and poisonous substances are only a small part of dangerous things. For example, computer programs - don't they come in all kinds? and who knows all they can do? And you can buy them for money, too. Well, that's it. Money - real money for which you can buy everything - should be taken from the commoner. In toto. And he should be given something else, simpler. Toy money with which he can only buy safe things, toys, and a little pleasure. …
How? Well, just like this.
We already got used - in Russia too - to such a small but pleasant little thing as bonus points, loyalty card crediting and other such niceties. Should you come to the tavern seven times, you will get a mug of free beer, a cookie or a discount from the main account on the eighth. These little toy moneys, at this point they are mainly utilized to goad the client into spending the real money. …
Consider: someone is issuing (why and by what right - let's omit these details for now) some conditional unit of exchange the "cost" of which is defined as 0.5 points of a card of some central Moscow store, plus 0.3 discount points of a popular restaurant chain, plus 0.2 points of a chain of women's clothing and perfume. It should be noted that money – roubles or dollars – is out of the question, only points. This conventional unit is called, say, "Sale". Sale is exchanged at trades – arranged as exchange currency trades – for other points, and (for the time being) is bought for money. And it is profitable: by buying Sales, you can buy more goods and services than for money directly. By three percent, not more - but they won't hurt, right?
Now imagine that you are offered to cover part of the price with Sales, on favorable terms. You save five percent, or even whole ten percent. However, you will be able to shop only there where Sales are accepted. But you it's not like you need anything else. You need to eat, get dressed, go to the cinema... you can get all this for Sales. And real money will be needed to pay taxes and for communal housing.
Why such difficulties? Little by little, one travels far. Dangerous toys - dollars, rubles, euros - should be taken away from people gradually, and in such a manner that they themselves give them up. In the end, the toy "Sales" will actually cover the basic needs of an ordinary citizen, and even ten percent above that. Do you insist on getting paid in real money? You will not be refused. But there will be rumors that you are saving money for something bad. Maybe for drugs? Or are you sending it to terrorists? Your account might begin to attract interest. And you personally, too. No, no, no, if you have nothing illegal, no one will bother you. Why do you need money anyway? Oh, you're saving for a diamond ring for your loved one? Well, we didn't foresee that, so we'll suggest to the jewelers that they put special points on diamonds and include it in the Sale basket.
It's not like they haven't started thinking along similar lines, however.
DaseindustriesLtd
late version of a small language model
3yr ago
To be clear, I do not fear Eliezer Yudkowsky or any other doomer. I think that for all the messianic ambition they are chiefly neurotic fools whose narratives are promoted and mainstreamed by people who've long been interested in what our departed friend JB called «power singularity» – just for good measure, to protect children and terrorists and such, you know the script; and who in turn may be the relatively well-intentioned party here.
Noted.
DaseindustriesLtd
late version of a small language model
3yr ago
Yes, I'm doing what I… feel like doing to preclude that outcome, although it is suboptimal to waste effort in terms of remaining expected utility, by my count. Now, I'm not a utilitarian. But normal influences of reward and punishment do mold my actions, so I don't feel like doing a whole lot.
The go-to pivotal action to avoid torture, on the other hand, is obvious enough and reliable. I don't put much stock in the Basilisk.
DaseindustriesLtd
late version of a small language model
3yr ago
Biology isn't magic, viruses can't max out all relevant traits at once, they're pretty optimized as is. I find the idea of superbugs a nerdsnipe, like grey goo or a strangelet disaster, a way to intimidate people who don't have the intuition about physical bounds and constraints and like to play with arrow notation.
(All these things scare the shit out of me)
Yes we can make much better viruses, no there isn't such an advantage for the attacker, especially in the world of AI that can rapidly respond by, uh, deploying stuff we already know works.
DaseindustriesLtd
late version of a small language model
3yr ago
For the last few years I assume I will likely get a permanent torture sentence in The World To Come. Centralized post-scarcity implies retribution against Evil People is cheap and desirable.
I live with it because life isn't supposed to make sense.
Consistent Agents are Utilitarian: If you have an agent taking actions in the world and having preferences about the future states of the world, that agent must be utilitarian, in the sense that there must exist a function V(s) that takes in possible world-states s and spits out a scalar, and the agent's behaviour can be modelled as maximising the expected future value of V(s). If there is no such function V(s), then our agent is not consistent, and there are cycles we can find in its preference ordering, so it prefers state A to B, B to C, and C to A, which is a pretty stupid thing for an agent to do.
But... that's how humans work? Actually we're even less consistent than that, our preferences are contextual so we lack information to rank most states. I recommend Shard Theory of human values probably the most serious intropection of ex-Yuddites to date:
A shard of value refers to the contextually activated computations which are downstream of similar historical reinforcement events. For example, the juice-shard consists of the various decision-making influences which steer the baby towards the historical reinforcer of a juice pouch. These contextual influences were all reinforced into existence by the activation of sugar reward circuitry upon drinking juice. A subshard is a contextually activated component of a shard. For example, “IF juice pouch in front of me THEN grab” is a subshard of the juice-shard. It seems plain to us that learned value shards are[5] most strongly activated in the situations in which they were historically reinforced and strengthened.
... This is important. We see how the reward system shapes our values, without our values entirely binding to the activation of the reward system itself. We have also laid bare the manner in which the juice-shard is bound to your model of reality instead of simply your model of future perception. Looking back across the causal history of the juice-shard’s training, the shard has no particular reason to bid for the plan “stick a wire in my brain to electrically stimulate the sugar reward-circuit”, even if the world model correctly predicts the consequences of such a plan. In fact, a good world model predicts that the person will drink fewer juice pouches after becoming a wireheader, and so the juice-shard in a reflective juice-liking adult bids against the wireheading plan! Humans are not reward-maximizers, they are value shard-executors.
This, we claim, is one reason why people (usually) don’t want to wirehead and why people often want to avoid value drift. According to the sophisticated reflective capabilities of your world model, if you popped a pill which made you 10% more okay with murder, your world model predicts futures which are bid against by your current shards because they contain too much murder.
@HlynkaCG's Utilitarian AI thesis strikes again. Utilitarianism is a strictly degenerate decision-making algorithm because it optimizes decision theory, warps territory to get good properties of the map, it's basically inverted wireheading. Optimizer's curse is unbeatable, forget about it, an utilitarian AI with nontrivial capability will kill you or come so close to killing as to make no difference; your life and wasteful use of atoms are inevitably discovered to be a great affront to the great Cosmic project $PROJ_NAME. Consistent utilitarian agents are incompatible with human survival, because you can't specify a robust function for a maximizer that assigns value to something as specific and arbitrary and fragile as baseline humans – and AI is a red herring here! Yud himself would process trads into useful paste and Moravecian mind uploads manually if he could, and that's if he doesn't have to make hard tradeoffs at the moment. (I wouldn't, but not because I disagree much on computed "utility" of that move). Just read the guy from the time he thought he'll be the first in the AGI race. He sneeringly said «tough luck» to people who wanted to remain human. «You are not a human anyway».
Luckily this is all unnecessary.
the space of minds is vast, much vaster than the instrumental convergence basin
DaseindustriesLtd
late version of a small language model
3yr ago
– Have you heard anything about the Westphalian secret court?
Sasha was a little taken aback. Then he thought about it.
– I don't really remember German history, – he admitted. – I've heard something, but that... Some kind of secret society like the Freemasons.
– Freemasons are Freemasons. - Wietz made himself comfortable, obviously intending to speak at length, - I'm talking about the Westphalian, aka Free or Femean Court. The name comes from the Latin fama, «popular opinion». The secret courts arose in the thirteenth century, in a situation similar to what we have now in Russia. Power exists, justice does not. Imperial justice did not work and did not want to work. But the Germans, unlike the Russians... – here Witz was silent for a second, and Sasha thought about how unpleasant this word, "Russians", was – took matters into their own hands. The Feme system worked like this: any Christian could turn to the secret court, and it summoned the accused to its session. If he appeared, he was tried according to the ancient laws and sentenced. If he did not appear, he was considered guilty, and the assassins caught up with him.
– Could a Jew turn to it? – Tsypin asked for some reason.
– He could turn, – Witz said; as one could describe it, pointedly.
– Sounds like the Mafia, – remarked Sasha.
Vincent wrinkled his forehead, trying to find the exact wording.
– It's Mafia that's like a Femean court. That's what it was, but then the uomini d’onore tried to switch sides, lost everything, and turned into mediocre mascalzoni. Fema is a different matter.
<...>.
For two centuries the Fema courts acted in isolation. But then came the idea of uniting under a single will.
– The party is a hand of million fingers, clenched in one smashing fist, – Sasha couldn't resist another stilted quote. He became somewhat animated: the conversation became bookish, and bookish talk calmed him down.
– You guessed it again, – the guard looked at Sasha with approval. – Exactly a fist. Latin for faustus. The Fema Courts, as well as a number of Fereins and some other forces, created an organization by that name. Which subsequently assumed responsibility for the order and prosperity of Europe.
http://haritonov.wiki/Вечность
DaseindustriesLtd
late version of a small language model
3yr ago
DaseindustriesLtd
late version of a small language model
3yr ago
Tons of great ideas.
Optimize hard or GTFO
A lot of shit on OpenPhilanthropy, FTX and EA.
How to host an autarkic/uncensorable site.
Why are we not better, harder, faster, stronger
Updating in the face of anthropic effects is possible
Seems more useful though cultural stuff is probably more interesting.
I reject that I need to prove something as logically impossible to ward off Yud's insistence that it's inevitable and justifies tyranny. This is sectarian bullshit and I'll address it in the text if I ever finish it. I think it's very relevant that his idea of proper scientific process is literally this:
Jeffreyssai chuckled slightly. "Don't guess so hard what I might prefer to hear, Competitor. Your first statement came closer to my hidden mark; your oh-so-Bayesian disclaimer fell wide... The factor I had in mind, Brennan, was that Eld scientists thought it was _acceptable_to take thirty years to solve a problem. Their entire social process of science was based on getting to the truth eventually. A wrong theory got discarded _eventually_—once the next generation of students grew up familiar with the replacement. Work expands to fill the time allotted, as the saying goes. But people can think important thoughts in far less than thirty years, if they expect speed of themselves." Jeffreyssai suddenly slammed down a hand on the arm of Brennan's chair. "How long do you have to dodge a thrown knife?"
...
"Good! You actually thought about it that time! Think about it every time! Break patterns! In the days of Eld Science, Brennan, it was not uncommon for a grant agency to spend six months reviewing a proposal. They permitted themselves the time! You are being graded on your speed, Brennan! The question is not whether you get there eventually! Anyone can find the truth in five thousand years! You need to move faster!"
"Yes, sensei!"
"Now, Brennan, have you just learned something new?"
"Yes, sensei!"
"How long did it take you to learn this new thing?"
An arbitrary choice there... "Less than a minute, sensei, from the boundary that seems most obvious."
"Less than a minute," Jeffreyssai repeated. "So, Brennan, how long do you think it should take to solve a major scientific problem, if you are not wasting any time?"
Now there was a trapped question if Brennan had ever heard one. There was no way to guess what time period Jeffreyssai had in mind—what the sensei would consider too long, or too short. Which meant that the only way out was to just try for the genuine truth; this would offer him the defense of honesty, little defense though it was. "One year, sensei?"
"Do you think it could be done in one month, Brennan? In a case, let us stipulate, where in principle you already have enough experimental evidence to determine an answer, but not so much experimental evidence that you can afford to make errors in interpreting it."
Again, no way to guess which answer Jeffreyssai might want... "One month seems like an unrealistically short time to me, sensei."
"A short time?" Jeffreyssai said incredulously. "How many minutes in thirty days? Hiriwa?"
"43200, sensei," she answered. "If you assume sixteen-hour waking periods and daily sleep, then 28800 minutes."
"Assume, Brennan, that it takes five whole minutes to think an original thought, rather than learning it from someone else. Does even a major scientific problem require 5760 distinct insights?"
"I confess, sensei," Brennan said slowly, "that I have never thought of it that way before... but do you tell me that is truly a realistic level of productivity?"
"No," said Jeffreyssai, "but neither is it realistic to think that a single problem requires 5760 insights. And yes, it has been done."
This guy has done fuck all in his life other than read, and write, and think. He has never been graded by a mean professor, never been regularized by shame and inadequacy in a class of other bright kids, never stooped to empirical science or engineering or business or normal employment, never really grokked the difference between the map and the territory. He has an unrealistically, wildly inflated impression of how powerful an intelligence contorted into a Hofstadterian loop is. He has infected other geeks with it.
Recursive self-improvement doesn't work very well. Rationalists become cranks, AIs degenerate. As for better ideas, see around here. It is certain that we can improve somewhat, I think. In the limit, we will get an ASI from a closed experimental loop. That really is like creating a separate accelerated civilization.
But with ANNs, unlike Lisp scripts, it seems to require a great deal of compute, and compute doesn't just lie on the sidewalk. Yud thinks an AGI will just hack into whatever it wants, but that's a very sci-fi idea from 1990s; something he, I believe, dreamed to implement in the way already described – a singleton in the world of worthless meat sacks and classical programs. If you hack into an AWS cluster today to do your meta-learning training run, you'll suspend thousands of workloads including Midjourney pics and hentai (that people …check in real time), and send alarms off immediately. If you hack into it tomorrow, you'll get backtracked by an LLM-powered firewall.
No, I'm not too worried about an orthodox Yuddite self-improving AI.
DaseindustriesLtd
late version of a small language model
3yr ago
Yes he does, his «I'll totally take it» scenario is AGI killing no more than 1 billion people, and he assigns 90% probability to AGI wiping out literally all life.
All of Ord's calculations seem to assume highly efficient coordinated conspiracy running the world instead of the slowly decaying shitshow one should expect from post-collapse civilization, but I guess that's par for the course.
Why do you assume GOFAI or uploads are safer? From what I can tell, this sort of reasoning is only grounded in aesthetic distaste for statistics, on in mushy intuitions like «much weight, very black box, there be shoggoth» like that fraud Leahy implies. Well, we can sparsify models and make them structurally simpler, it seems, but what of it? The causal structure of a mind worth the name cannot fit into human consciousness, and it's improbable that there are theorems that give sufficiently secure bounds for its qualitative behavior; if there are, they could as well hold for NNs, because why the hell not, they are differentiable functions. GOFAI either doesn't work at all (there are good reasons to expect it to never work, or to require neural nets to assemble) or is more risky in terms of having sharp transitions and giving us unbounded optimizers. Indeed, all of Yud's fears were invented basically for GOFAI or are paradigm-agnostic; he never expected NNs to work, not without intractable compute expenditure on the scale of simulating historical evolution, and so the question of their – now supposedly inevitable – misalignment was moot. Likewise I don't see what makes uploads inherently safe but doesn't hold for NNs. Are they too fat to be a threat or what? They'd still be data, still amenable to self-editing, still capable of accelerated and inscrutable interactions, eventual economic superiority over baseline humans, inhumanity. See Pantheon, it's as good a doomsday narrative as any.
Reminder: having received millions of dollars in donations, Yud has turned his AI safety organization MIRI into a sinecure for mathletes preoccupied with weird self-referentiality studies and esoteric decision theories. What little alignment-related investigations they had were nonsensical to the point of casting serious doubt on coherence of their conceptual apparatus:
Insane asylum stuff. Unsurprisingly, Christiano, ever the gentleman, describes the situation such:
No, really, what do you have against neural networks? I think it's tremendous luck in terms of safety that we got them to work and can approximate/interpolate/extrapolate desirable behavioral properties directly, instead of languishing for a few more centuries and praying that our Enlightened Infra-Bayesian Lords didn't make an oopsie with the sign somewhere and covered it up like another Leverage, or the Wuhan affair.
More options
Context Copy link