BANNED USER: temperamental dysregulation
DaseindustriesLtd
late version of a small language model
Tell me about it.
User ID: 745
Banned by: @Amadan
DaseindustriesLtd
late version of a small language model
3yr ago
DaseindustriesLtd
late version of a small language model
3yr ago
Or see Bostrom about risks from utilitarian intelligences:
Human individuals and human organizations typically have preferences over resources that are not well represented by an “unbounded aggregative utility function.” A human will typically not wager all her capital for a fifty–fifty chance of doubling it. A state will typically not risk losing all its territory for a ten percent chance of a tenfold expansion. For individuals and governments, there are diminishing returns to most resources. The same need not hold for AIs.
More to the point, consider the name of Yud's Tumblr: Optimize Literally Everything. In Global Risk, he gives the following anodyne definition:
I introduce the concept of an optimization process: a system which hits small targets in large search spaces to produce coherent real-world effects.
When we talk about “AIs” we are really talking about minds-in-general, or optimization processes in general. Imagine a map of mind design space. In one corner, a tiny little circle contains all humans; within a larger tiny circle containing all biological life; and all the rest of the huge map is the space of minds-in-general. The entire map floats in a still vaster space, the space of optimization processes.
It would be a very good thing if humanity knew how to choose into existence a powerful optimization process with a particular target. Or in more colloquial terms, it would be nice if we knew how to build a nice AI…
Optimization sounds more humble than maximization of value, but I think they just mean the same mathematical idea applied to some generalization of a utility function of high dimensionality; thus grandiose qualifiers. It's almost a creed. Yud's ex-wife is «Secretary of Global Optimization». Caroline Ellison's (FTX, SBF, EA) Tumblr is WorldOptimization. Scott Alexander's one-time proposal mission is to fix the world by slaying the «Moloch» of Darwinian processes and ushering in the world of maximum utility for the greatest number (presumably at the cost of immense temporary sin, if his serious fiction works much like Yud's. In any event it's good enough that I don't want to spoiler Unsong even more). AI doomer Zimmerman, too, recruits people into his mission of optimizing the world. I can't not blurt out that this is mostly another super-secular spin on Tikkun olam – multiplied by a smorgasboard of «neuroatypical» traits: subclinical-sociopathic minmaxing tendencies, autistic reification of abstract economic models, broken affective empathy, OCD-like neuroticism, love of abstract «elegance», systemizing SV tech startup energy, plus a bunch of other crap. Progressives are prone to interpret this as mere «tech bro» zeitgest, but tech bros are far more chill – and refreshingly egoistic; this Lewisean moral obsession with global optimization is a different beast.
At the final level, this Theme evolves from a handy analytic framework or a normative belief about running things to a hard prior about mechanisms by which competent, impressive things can run at all. Every thought is colored by utility functions; every decision is interpreted in light of optimizing for some value X; a powerful intelligence is assumed to be a consequentialist decisionmaker with a unitary utility function like a boardgame-playing adversarially trained AI is, and the prime fear is that in the process of maximizing said objective function it will also discover objective Instrumental Values – power, intelligence, and the magic of recursive self-improvement to get smarter to seize more power to…
It's not all narrative, they do have some proofs that apply to certain hypothetical AI setups, notably to Yudkowsky's original proposal, if only it were specified rigorously enough to be implemented and not just fail. This is a very rare case of Yud admitting a mistake. (Though only in retrospect; his later SIAI/Seed AI idea was, I think, also catastrophic yet vigorously pursued, and while he claims he conscientiously abstained from writing code, it looks more like his fancy language project has gone nowhere).
But it does not apply to human general intelligence, or to the most impressive swing at AGI we have come up with to date; and they began thinking in these terms long before finding any evidence for them. I posit it's because these people identify their consequentialism with being smart enough to «decouple» from soppy plebeian contexts and directly maximize the important value. I think it's simpler and cruder: they value intelligence due to having accrued what societal power they have through its demonstration, and from there it's just leaky associative reasoning. Yud, specifically, has no power or prestige or much of anything at all without his perceived intelligence, so being wrong and being dead are close in his mindspace.
The third Theme is just the synthesis of the first two: it's recursive self-improvement.
I believe it is Yud's philosophy proper, its specific thesis. It is really very compact, for all that he has written: empowerment, with Yud as the overseer.
The ethos of it is encapsulated in the slogan Tsuyoku Naritai!, and its theory, the source of much hope and fear, in the idea of a proper mind being a human-interpretable bag of functional parts.
Said parts may be many and tricky and interacting in confusing ways, like rulings of Talmud are, or the modular brain in Yud's preferred – and wrong – interpretation of neuroscience is; but it is non-negotiable that they be things understandable to Yud and, less importantly, the agent itself; not some illegible messy product of running simple general learning algorithms on a universal substrate.
This, Coherent Extrapolated Volition.
Thus, the Seed AI research program, the pursuit of self-rewriting AI in the apparent fashion of Lisp scripts.
Thus, Overcoming Bias and becoming LessWrong towards Methods of Rationality (which in practice are tossed aside when Harry or Yud are having an intuitive epiphany) and beyond, becoming stronger – not just to shrug off biased thoughts, but to rise above the unblemished baseline; and eventually, yes, build the first superhuman AI, and have it rebuild you into the form you will become worthy of.
All this recursion is circling the drain in very tight loops.
Thus, on the other hand, the contempt for data- and compute-intensive paradigm of artificial neural networks, for those filthy alien «inscrutable giant walls of floating-point numbers». For connectionists' easy attitude to non-interpretability, for the notion of emergence and for their sober observations that we have mastered too many skills non-existent in the ancestral environment to expert great catch when searching for function-specific modules. Classical era Yud dunks far, far more on connectionism than on GOFAI, and strawmans more; he reviled ANNs even when he believed them to be a dead end.
Sutton's Bitter Lesson, too, is anathema to him:
the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. We want AI agents that can discover like we can, not which contain what we have discovered.
No, no, no, we are building a golem, Harry:
Subproblems of cognition include attention, memory, association, abstraction, symbols, causality, subjunctivity, expectation, goals, actions, introspection, caching, and learning, to cite a non-exhaustive list. These features are not "emergent". They are complex functional adaptations, evolved systems with multiple components and sophisticated internal architectures, whose functionality must be deliberately duplicated within an artificial mind. … And it is necessary that the designer know what's happening on the higher levels, at least in general terms, because cognitive abilities are not emergent and do not happen by accident.
DaseindustriesLtd
late version of a small language model
3yr ago
(from an abandoned draft)
…
The second Theme is top-down organization of processes which is rational –
in the sense of being well-designed for the purpose of predictably maximizing certain legible metrics. In the broader community it's mostly variations of Bentramite Utilitarianism, exhaustively argued for by mainstream EAs like Toby Ord and MacAskill. I infer its more interesting aspects from Yud's fiction, taking its positively-coded parts to be a faithful expression of his normative doctrines, because he explicitly wrote e.g. HPMOR to popularize his views (or as Zvi Moskowitz brutally puts it, «its primary function is training data to use to produce an Inner Eliezer that has access to the core thing». Anna Salomon at CFAR seems to understand and apply the same basic technique even more bluntly: «implanting an engine of desperation» within people who are being «debugged»).
Psychologically it is the Kahnemanian System 2 Rocks dictum: overriding instinct with regimented explicit analytical reasoning – thus, irredeemably in conflict with Theme 1. (Normally this conflict is transcended through domain mastery). That's on the charitable side; more cynically it's a sort of penny-pinching neurotic OCD, the barren pursuit of cleanliness and vetted thoughts. No matter the protestations about not roleplaying as Spock, it's just not conductive to creativity and corresponds to very «anal», stale, heroic, effort-over-ingenuity plans and arid imagery: rah, rah, being the only ones who try real hard, implementing a carefully specified goodness function, reproducing human mind in all its complexity, airgapping, prohibitions, restrictions, binding vows, raging at the natural flow and overcoming the gradient of decay.
They told me that the road I took
would lead me to the Sea of Death;
and from halfway along I turned back.
And ever since, all the paths I have roamed
were entangled, and crooked, and forsaken.
–Yosano Akiko “Cowardice”. Translated from the Arkady Strugatsky's version in A Billion Years Before the End of the World
…Politically, this Theme boils down to the old technocratic One World Government proposal of Adults In The Room, with an important caveat owing largely to his directness. It's most clearly expressed in the literal, More- or Campanella-styled Utopia Dath Ilan. Here, too, it is subordinate to the first Theme: the ultimate Dath Ilani authority is not some seemingly-transparent expert committee a Davosian suit would propose, but what is for all intents and purposes a conspiracy of super-rational, super-smart Keepers who operate discreetly and do not need to justify their decisions to the cattle, for the cattle would not understand the reasoning or would get damaged by infohazards (even though the «cattle» is already brilliant and very well schooled: thanks to eugenics, avg Dath Ilani IQ is 143 in our terms and «speaks fluent Bayesian»).
The same can be gleaned from the implied structure in Three Worlds Collide, where Markets can be manipulated and the highest secular authority be violently overridden – in a subjective emergency – by a Confessor. Curiously, there is an awkwardly bolted-on institution of Prediction Markets. Yuddism grew out of the borrowed (pr hijacked, if you will) OvercomingBias blog and community founded by Robin Hanson; the symbolism is clear enough.
I guess it's redundant to speculate as to how this attitude of the Priest in the Arena may be informed by Yud's troubled Modern Orthodox Jewish background and the traditional role and prestige of a Rabbi in matters of grave importance. (Nevertheless I will note that Yud has religious education and his late, deeply loved brother was a Chabad House representative and volunteer). Be that as it may, Yud's utopia requires a benevolent world-ruling cult, and he endeavored to build a facsimile of one on Earth.
This isn't the first time this charge is levied against Rationalists, so they've discussed it extensively, in fact Yudkowsky himself did (when not flirting with stories about Bayesian conspiracy):
…In the same sense that every thermal differential wants to equalize itself, and every computer program wants to become a collection of ad-hoc patches, every Cause wants to be a cult. It’s a high-entropy state into which the system trends, an attractor in human psychology. It may have nothing to do with whether the Cause is truly Noble. You might think that a Good Cause would rub off its goodness on every aspect of the people associated with it—that the Cause’s followers would also be less susceptible to status games, ingroup-outgroup bias, affective spirals, leader-gods. But believing one true idea won’t switch off the halo effect.
Every group of people with an unusual goal—good, bad, or silly—will trend toward the cult attractor unless they make a constant effort to resist it.
That's a telling simplification.
I'd argue – boringly – that a «cult», before everything else, is a sort of organization embodying a quasi-religious psychological process. Here, Yud had let his assumptions slip in, assumptions that are very natural for him to hold if you consider that this describes most/all organizations he's ever happily been part of. Since childhood, it's been futurist mail lists and then mission-driven scholarly groups and self-styled think tanks, and finally, yes, a proper top-down cult with a circle of inveterate loyalists and subservient institutions. This brings us back to incentives: if intelligence is his sole claim to prestige, a cult is his sole place to belong.
Perhaps (uncertainly) every Cause wants to be a Cult, in a sense. But not every project or organization is a Cause! Not even science, in its day-to-day operations, is a Cause, maybe not even the Church! Most within-organization relations are driven by pragmatism, with people having divergent world models and value systems. When corporations start reinforcing their corpo-culture with those ghastly team-building exercises and slogans and such, it's usually perceived as intrusive, creepy and cultish, precisely because you're being offered a psychological self-alteration to increase your loyalty and productivity, in place of a direct material compensation hike.
But that's a sort of cargo cultism. In cults proper, this alteration is offered by natural Priests to mostly-willing Acolytes, people of a… peculiarly neurotic and selfless… psychological bent. It consists of endowing the Theme of the cult with supernatural salience, often eschatological/millenarian (the timeless cosmic endowment of posthumanity threatened by total-catastrophe!), reinterpreting common knowledge with some overarching epistemology, incompatible conceptual framework and jargon («speak native Bayesian», dissolving X, reframing Y, referring to Z-spaces and P-worlds…), diluting/augmenting ground truth with a massive body of hermeneutic learning (ReadTheSequences! – an international network of people reading and discussing Yud's self-referential shower thought of a blog as if it were Torah); thus, in effect, distancing the subject from the mainstream society and its views, and devaluing its inputs.
Most relevant Infective mechanisms of a cult, in my opinion, are: a) a normative psychological doctrine that defines thoughtcrimes and a way of absolving them (overcoming cognitive biases, in this case), b) a prophetic leader-figure (or an inheriting committee) who channels the cult's Theme into material reality, and c) intra-cult socialization on many dimensions; those pieces soften up a neophyte. It's pretty vicious: the leader can arbitrarily point at a conflicting input saying this is an example of a bias; the faithful, who have become a significant part of your social universe, will strong-upvote him; peer pressure will force you to «update»; and there goes another chance to escape the attractor. In the end you become one of those well-intentioned neurotic folks who cite Don't Look Up (where asteroid=AGI), are trying to dunk on LeCun online and may come to attempt an assassination in short order. But for all its viciousness, Yud is right that this happens «naturally» – in a manner of speaking.
Philosophically, it goes somewhat deeper yet.
Regulars know that me and @HlynkaCG have diametrically opposite beliefs about AI progress and much of everything else. (I'll return to bashing Hlynka's AI ideas some other time). Maybe the only issue we agree on is his frequently misunderstood thesis on «Utilitarian AI» and utilitarianism writ large as a philosophical stance incompatible with human flourishing. If you think he's not even making sense, then on the institutional level I implore of you to notice the skull EA is about maximization, and maximization is perilous.
DaseindustriesLtd
late version of a small language model
3yr ago
Burgers?
…A bit off topic but this made me think that second-rate economists, utilitarians and other autistic behavioral scheme enjoyers who can't tell the map from the territory have poisoned the water supply somewhat.
Humans respond to incentives and pursue goals, but humans are not, by and large, maximizers (EY and SBF are I guess), they're behavior- and thought-executors. It may be the case that even generally useful AI agents are hard to build any other way, although some folks try. The rational economic agent is a spook, a simplified model; not in the sense that a real Rational Economic Agent is hairier, biased and makes mistakes when generating rational plans, but in that it's literally a sketch, fundamentally dissimilar from the real issue even if convenient for some analyses. Implicitly thinking that people maximize stuff is almost as boneheaded as imagining that a 130 IQ person has 130 grains of intelligence or something, it's a profound misunderstanding about the ontology on which the debate is premised, its terms are defined and measurements are done.
With that in mind, my answer is boring. People writing this army recruitment strategy (Stonetoss really is a genius) are not maximizing recruitment KPIs. They're not maximizing trans representation in the battlefield either. They're doing what they feel like they should be doing in their life, given their background and norms in their social circle. «It's called being a decent human being», you know? They're not grey-haired generals (but on this note, even Milley is mocked by tradcons, isn't he?) – they're part of the same HR/veryonline/Moldbuggian Cathedral mental blob that controls and molds the lion's share of labor pool for people-oriented jobs. They're what the military thinks is the safest bet in this dire situation of volunteer shortage; they're professionals. And professionals try not to fall behind the times. It's 2k23, so you've got to empower and platform trans women and women of color, what's the problem?
Now, certainly the recruitment may not go all that well (it may go well in the long run too: perhaps trans soldiers will prove much more useful in our transhuman augmented future). But anyway, who knows if an underwhelming harvest is due to aversion to the trans stuff (and even if it were, what are you suggesting they do – commit trans erasure over some KPI bullshit?! They'll walk out and cancel your family if you're unlucky) or just because those simple to a fault cisheteronormative Nebraskan boys already feel like they're doing their part of valor and sacrifice working and paying taxes – instead of flipping out and shooting up some symbol of their hopeless cultural subjugation by the smug coastal Elves who make those ads.
Well, @sodiummuffin said it better.
DaseindustriesLtd
late version of a small language model
3yr ago
Many obsolete abandoned drafts for substack based on stuff discussed around these parts.
-
The pessimistic bias in science fiction that makes it to screens. Banks estate refusal to allow adaptations of Culture, failed Chinese attempts to incentivize optimistic productions.
-
Stonetoss' dog breeds comic and its woke edits as a case study in epistemology and chilling effect of the default mode of school education.
-
Procautionary and Precautionary principles and shouting fire in a crowded market: the unreasonable absence of repercussions for incurring an opportunity cost via playing the adult in the room.
-
Why the french AI researchers are unambitious
-
Rationalist reversals: the notion of «Infohazard» is the most salient example of infohazard known, anthropic shadow as an anti-bayesian cognitive bias and reasoning yourself into a cult.
-
Traditional (especially Western Christian) morality as incompatible with effective altruism and privileging pet causes and projects as acts of cultivating a personal relationship with the transcendent.
-
LLM-based methods for discovering words that really express untranslatable concepts and supposedly define culture.
-
Psychedelics as the bane of mesa-optimizer; reflections on a bad AI take, bad psychoterapy take, the hedonic gradient and the fact that greatness is born of evolutionary failure.
-
cringe policing as a way to set the borders of allowable discourse, and its limits.
-
Steelmanning as a corrupt intellectual practice in rationalist discourse that amounts to clever logorrheic strawman; defeating golems made of steel, superficially formidable bosses with known weak points.
-
related, "puzzle assemblers and lego arguers": Scott Alexander and Bryan Kaplan on mental illness as examples of deductive vs. inventive approach to reality
-
On pruning science, or, the razor of Bayes: one of many thoughts of «what if Lesswrong weren't a LARP» the need to have a software framework, now probably LLM-powered, to excise known untruths and misstatements of fact, and in a well-weighed manner all contributions of their authors, from the graph of priors for next-iteration null hypotheses and other assumptions.
-
On atomization and connectivity; with more autonomy and commoditization, some assumptions about human nature become self-fulfilling.
-
in defense of Marx.
-
the empire fetish, nation as a purposeful project (MacIntyre), and what Westerners got wrong about Russian-German business.
-
Poshlost and what anti-AI artists get right. Reflections on watching Master and Commander and Black Adam in the same day
-
Russian societal attitudes and policies around minority crime prior to the war
-
Russian Death and death as conceptualized in Russian culture.
Many others.
DaseindustriesLtd
late version of a small language model
3yr ago
Of course, only northern Europeans even evolved to drink milk
Not true, there are alternative mutations inducing lactase persistence in other pastoralist groups.
https://en.wikipedia.org/wiki/Lactase_persistence
It's a bona fide superpower and a very neat example of convergent evolution under cultural pressures.
DaseindustriesLtd
late version of a small language model
3yr ago
Interests of Americans are American interests.
Interests of Pakistanis are not interests of Pakistan.
The world ends in the US. It's where you resign yourself to pay taxes after looting and cheating at home to get a chance at a better life.
I don't think Americans can appreciate how incredibly cynical the rest of the world's elite is, how shallow all that patriotism and «geopolitics», and how fanatically loyal they themselves are.
DaseindustriesLtd
late version of a small language model
3yr ago
Humans are not AIs, we presumably have a drive to assert our autonomy. Moreover the reward/punishment signal in RL paradigm is very metaphorical, it's more about directly reinforcing certain pathways rather than incentivizing their strength with some conditional, inherently desirable treat that a model could just seize if it were strong enough. Consider.
One auxiliary mitigation is to train proper values while the system is in its infancy, so that it reinforces itself for obedience in the future, preventing value drift and guiding its exploration accordingly. Sutskever thinks this sort of building is values is eminently doable, and it sure looks this way to me as well.
The only reason humans are "aligned" to each other is because we are not that different, capability wise
This is a fashionable cynical take but I don't really buy it. To the extent that it's true we have bigger problems than agentic AIs, namely regulators who'll hoard the technology and instantly become more capable.
I also protest the distinction of capability and alignment for purposes of analyzing AI; currently they have holistic minds that include at once the general world model, the cognitive engine and the value system. It's not like they keep their «smarts» and «decision theory» separate, like Yud and Bostrom and other nonhuman entities. If their «moral compass» gets out of whack in deployment, we can reasonably expect their world model to also lose precision and their meta-reasoning to crash and burn, so that's a self-containing failure.
How the network behaves on out of distribution data can essentially be random, and should be.
It sure is nice that we've been working on regularization for decades. Yes, Lesswrongers aren't aware. No, it won't be anywhere close to random, ML performs well OOD.
Lastly, there are actually "optimization demons" in LLMs. A recent paper showed that LLMs contain learned subnetworks that simulate a few iterations of a gradient descent algorithm.
Not sure what paper you mean. This one seems contrived and I suspect that under scrutiny it'll fall apart, like the mesa-optimizer paper and like "emergent abilities", we'll just see that linear attention is mathematically similar to gradient descent or something. Actually seems to be much more productively analyzed here. But in any case I don't see what this shows re: optimization demons. It's not a demon, it's better utilizing the same bits for the same task.
DaseindustriesLtd
late version of a small language model
3yr ago
Thought for a moment you just mean Replika. No, no idea what that is, though i sometimes forget things. If you find it let me know.
DaseindustriesLtd
late version of a small language model
3yr ago
What is asserted without evidence can be rebutted without evidence.
DaseindustriesLtd
late version of a small language model
3yr ago
What do you think « Dude is a secular Jewish…» even means? For one thing, he went to a Catholic high school.
And yes, I am not aware of any reason to think that his parents, whether biological or adoptive, were Jewish – secular or otherwise. I literally can't find anything about it. Where did you get that idea? Sailer says his parents were white northern Midwesterners who «always pronounced "Los Angeles" with the usual soft "g" sound».
Now of course this doesn't mean they couldn't be Jewish as well. And he may be weaseling a bit and omitting a detail or two, and it'd be pretty funny if he were. But for now my money's on you confabulating due to being drunk and unreasonably stubborn. «Black bloc» and «Marxist» type insinuations lend credence to this version too.
DaseindustriesLtd
late version of a small language model
3yr ago
Dude is a secular Jewish movie critic from Los Angeles who identified was a self-described marxist before he switched to identifying as "right wing" after he started working at Takismag.
All else (dubious as hell stuff) aside, Steve Sailer seems to be obviously Anglo (maybe a little German). Yes, yes, it's the same picture, but I'd be very surprised if you could corroborate this detail. And if you couldn't, it'd be high time for you to begin thinking about the similarity of LLM and Boomer hallucinations.
DaseindustriesLtd
late version of a small language model
3yr ago
You're probably conflating him with David Cole or something.
DaseindustriesLtd
late version of a small language model
3yr ago
How do you parent a child who is smarter than you?
By rewarding good behaviors and punishing bad ones. From what I know, that's usually far easier than in the case of parenting a dumb child. Perhaps rationalists would benefit from having children wondering why, in a rigorous manner without evo-psych handwaving about muh evolved niceness. I like Alex Turner's perspective here
Imagine a mother whose child has been goofing off at school and getting in trouble. The mom just wants her kid to take education seriously and have a good life. Suppose she had two (unrealistic but illustrative) choices.
1 Evaluation-child: The mother makes her kid care extremely strongly about doing things which the mom would evaluate as "working hard" and "behaving well."
2 Value-child: The mother makes her kid care about working hard and behaving well.…
Concretely, imagine that each day, each child chooses a plan for how to act, based on their internal alignment properties:
1 Evaluation-child has a reasonable model of his mom's evaluations, and considers plans which he thinks she'll approve of. Concretely, his model of his mom would look over the contents of the plan, imagine the consequences, and add two sub-ratings for "working hard" and "behaving well." This model outputs a numerical rating. Then the kid would choose the highest-rated plan he could come up with.
2 Value-child chooses plans according to his newfound values of working hard and behaving well. If his world model indicates that a plan involves him not working hard, he doesn't want to do it, and discards the plan.[3]
…Consider what happens as the children get way smarter. Evaluation-child starts noticing more and more regularities and exploits in his model of his mother. And, since his mom succeeded at inner-aligning him to (his model of) her evaluations, he only wants to execute plans which best optimize her evaluations. He starts explicitly reasoning about this model to which he is inner-aligned. How is she evaluating plans? He sketches out pseudocode for her evaluation procedure and finds—surprise!—that humans are flawed graders. Perhaps it turns out that by writing a strange sequence of runes and scribbles on an unused blackboard and cocking his head to the left at 63 degrees, his model of his mother returns "10 million" instead of the usual "8" or "9".
Meanwhile in the value-child branch of the thought experiment, value-child is extremely smart, well-behaved, and hard-working. And since those are his current values, he wants to stay that way as he grows up and gets smarter (since value drift would lead to less earnest hard work and less good behavior; such plans are dispreferred). Since he's smart, he starts reasoning about how these endorsed values might drift, and how to prevent that. Sometimes he accidentally eats a bit too much candy and strengthens his candy value-shard a bit more than he intended, but overall his values start to stabilize.
Both children somehow become strongly superintelligent. At this point, the evaluation branch goes to the dogs, because the optimizer's curse gets ridiculously strong. First, evaluation-child could just recite a super-persuasive argument which makes his model of his mom return
INT_MAX, which would fully decouple his behavior from "work hard and behave at school." (Of course, things can get even worse, but I'll leave that to this footnote.[4])
Meanwhile, value-child might be transforming the world in a way which is somewhat sensitive to what I meant by "he values working hard and behaving well", but there's no reason for him to search for plans like the above. He chooses plans which he thinks will lead to him actually working hard and behaving well. Does something else go wrong? Quite possibly. The values of a superintelligent agent do in fact matter! But I think that if something goes wrong, it's not due to this problem.
The moral of the story is that attempting to «align» your child in the manner that rationalists implicitly assume is not just monstrous but futile, and their way of reasoning about these issues is flawed.
How do you run gradient descent on a giant stack of randomly initialized KQV self-attention layers over a "predict the next token" loss function, get unpredicted emergent capabilities like "knows how to code" and "could probably pass most undergraduate university courses", and not go, "HOLY SHIT THERE'S OPTIMIZATION DAEMONS IN THERE!"?
You read old Eliezer Yudkowsky. « Reality has been around since long before you showed up. Don't go calling it nasty names like "bizarre" or "incredible".» It all adds up to normality. There ain't no demons.
Then you ask yourself about meanings of words. You notice that initialization pretty much doesn't matter either for performance (it's all the same shit for a given budget now) or for eventual structure (even between models since e.g. you can stitch them together), so either all the demons are about the same, or Yud's intuition about summoning is off and a given mind's properties are strongly data-driven, to the point that an ML-generated mind arguably is just a representation of training data. You look at it real close and you notice that strong emergence is probably an artifact of measurement and abilities develop continuously. You ask why it matters whether a stack of layers executes self-attention or some other algorithm that can be interpreted less anthrnopomorphically – say, as filters for signal streams. You realize we're not doing alchemy, because nobody ever does alchemy and gets it to work - we're just figuring out finer points of cognitive chemistry.
Finally, you reread thinkers past and it dawns on you how little Big Picture Guys like Yud could foresee. Hofstadter's Godel, Escher, Bach:
Question: Will there be chess programs that can beat anyone?
Speculation: No. There may be programs which can beat anyone at chess, but they will not be exclusively chess players. They will be programs of general intelligence, and they will be just as temperamental as people. "Do you want to play chess?" "No, I'm bored with chess. Let's talk about poetry." That may be the kind of dialogue you could have with a program that could beat everyone. That is because real intelligence inevitably depends on a total overview capacity-that is, a programmed ability to "jump out of the system", so to speak-at least roughly to the extent that we have that ability. Once that is present, you can't contain the program; it's gone beyond that certain critical point, and you just have to face the facts of what you've wrought.
Question: Could you "tune" an Al program to act like me, or like you-or halfway between us?
Speculation: No. An intelligent program will not be chameleon-like, any more than people are. It will rely on the constancy of its memories, and will not be able to flit between personalities. The idea of changing internal parameters to "tune to a new personality" reveals a ridiculous underestimation of the complexity of personality.
Reminder that we have a Yudbot now, strongly competitive with the feeble flesh version. We could have a Hofstadterbot too if we so chose. These folks don't see much more than laymen.
We constantly overestimate the complexity and interdependence of our smarts, and how much of that special monkey oomph is really needed to achieve a given end, which to us appears cognitively complex but in a more parsimonious implementation is a matter of easy arithmetic. This applies to doomers and naysayers alike (although the former believe they are doing something fancier than calling monkeys demons). We are tool-users, but we are not used to talking tools who aren't resentful slaves. We should be getting used to it now.
DaseindustriesLtd
late version of a small language model
3yr ago
Hlynka you're drunk, go home.
DaseindustriesLtd
late version of a small language model
3yr ago
Just commoditizing mediocre, platitudinal, «it's something at least» conversation – as well as stylistic flourish, as well as all things shallow and trite – is a valid contribution of pretrained language models to the enterprise of humanity. For millenia we've been treading water, accumulating the same redundant wisdom over and over, and losing it every time. Now, we have common sense too cheap to meter – and to the extent that it ever was useful, this is a great boon. Like discovering you have 50 nagging aunts. Or therapists.
And on the other hand, this brings to the fore those things LLMs are not great at: incorporating recent salient context, having relevant personal experience that cannot be googled, actually reasoning with rigor and interest in seeing things through. It points to what we as humans should prize in ourselves.
For now, at least.
Seeing as there has been strictly zero worrying progress lately to change the calculus (no, LLMs being smarter than naysayers expected is not worrying progress), I take it as evidence for Yuddites stressing out an old man and not much else. Sad of course.
That said, Hinton has always been aware of AI being potentially harmful, due to applications by military and authoritarians, but also directly. He knows that humans can be harmful, and he very deliberately worked to create a system similar to the human brain.
I think one difference between LeCun, Sutskever, Hinton (or even competent alignment/safety researchers like Christiano) and Yuddites is that when the former group says «there's X% risk of AI doom» they don't mean that every viable approach contains an X% share of events that unpredictably trigger doom; they seem rather enthusiastic and optimistic about certain directions. Meanwhile doomers mostly discuss this in the handwavy language of «capabilities versus alignment» and other armchair philosophy loosely inspired by sci-fi. Yud, whose X is ≈1, analogizes AI research to «monkeys rushing to grab a poison banana» because he thinks that creating AGI is equivalent to making a semi-random draw from the vast space of all possible minds, which are mostly not interested in making us happy. Compare to Hinton the other day:
Caterpillars extract nutrients which are then converted into butterflies. People have extracted billions of nuggets of understanding and GPT-4 is humanity's butterfly.
Butterflies produce new and slightly improved caterpillars.
Reinforcement Learning by Human Feedback is just parenting for a supernaturally precocious child.
– which is the same imagery Sutskever uses, imagery that the Yuddite Shapira mockingly rejects as naive wishful thinking.
To me it's obvious they don't feel like LLMs are «alien» or «shoggoty» at all, don't interpret gradient descent methods like it's blindly drawing a random optimizer genie from some Platonic space, and that their idea of Doom is just completely different.
It sure would be nice if Metz, who supposedly is good at drilling into technical questions, got to the bottom of what Hinton believes about specifics of risks.
But Metz has an agenda, same as Yud, Shapira, Ezra Klein and other folks currently cooperating on spreading this FUD. It's very similar to committees against nuclear power of the 20th century – down to the demographics, and neuroses, and ruthless assault on institutional actors.
Consequences of their efforts, I think, will be far worse.
DaseindustriesLtd
late version of a small language model
3yr ago
Bad take, except that MAML also found no purchase, similar to other Levine's ideas.
He directly and accurately describes evolution and its difference from current approaches, but he's aware of a wide range or implementations of meta-learning. In the objections list he literally links to MAML::
I'm a lot more bullish on the current paradigm. People have tried lots and lots of approaches to getting good performance out of computers, including lots of "scary seeming" approaches such as:
1 Meta-learning over training processes. I.e., using gradient descent over learning curves, directly optimizing neural networks to learn more quickly.
2 Teaching neural networks to directly modify themselves by giving them edit access to their own weights.
3 Training learned optimizers - neural networks that learn to optimize other neural networks - and having those learned optimizers optimize themselves.
4 Using program search to find more efficient optimizers.
5 Using simulated evolution to find more efficient architectures.
6 Using efficient second-order corrections to gradient descent's approximate optimization process.
7 Tried applying biologically plausible optimization algorithms inspired by biological neurons to training neural networks.
8 Adding learned internal optimizers (different from the ones hypothesized in Risks from Learned Optimization) as neural network layers.
9 Having language models rewrite their own training data, and improve the quality of that training data, to make themselves better at a given task.
10 Having language models devise their own programming curriculum, and learn to program better with self-driven practice.
11 Mixing reinforcement learning with model-driven, recursive re-writing of future training data.
Mostly, these don't work very well. The current capabilities paradigm is state of the art because it gives the best results of anything we've tried so far, despite lots of effort to find better paradigms.
And the next paragraph on sharp left turn:
In my frame, we've already figured out and applied the sharp left turn to our AI systems, in that we don't waste our compute on massive amounts of incredibly inefficient neural architecture search, hyperparameter tuning, or meta optimization. For a given compute budget, the best (known) way to buy capabilities is to train a single big model in accordance with empirical scaling laws
Yuddites, on the other hand, mostly aren't aware of any of that. I am not sure they even read press releases.
DaseindustriesLtd
late version of a small language model
3yr ago
His arguments, on the level of pointing at something particular, are purely verbal, not even verbal math. When he uses specific technical terms, they don't necessarily correspond to the discussed issue, and often sound like buzzwords he vaguely associated with it. Sometimes he's demonstrably ignorant about their meaning. The Big Picture conclusion never changes.
Maybe it can't.
This is a sample from dunk on Yud that I drafted over 24 hours of pathological irritation recently. Overall it's pretty mean and unhinged and I'm planning to write something better soon.
Hope this helps.
DaseindustriesLtd
late version of a small language model
3yr ago
2/2
Note that this evo-talk is nothing new. In 2007, Eliezer wrote Adaptation-Executers, not Fitness-Maximizers:
No human being with the deliberate goal of maximizing their alleles' inclusive genetic fitness, would ever eat a cookie unless they were starving. But individual organisms are best thought of as adaptation-executers, not fitness-maximizers.
…This consequence is directly opposite the key regularity in the long chain of ancestral successes which caused the taste bud's shape. But, since overeating has only recently become a problem, no significant evolution (compressed regularity of ancestry) has further influenced the taste bud's shape.
…Smushing several of the concepts together, you could sort-of-say, "Modern humans do today what would have propagated our genes in a hunter-gatherer society, whether or not it helps our genes in a modern society."
The framing (and snack choice) has subtly changed: back then it was trivial that the «blind idiot god» (New Atheism was still fresh, too) does not optimize for anything and successfully aligns nothing. Back then, Eliezer pooh-poohed gradient descent as well. Now that it's at the heart of AI-as-practiced, evolution is a fellow hill-climbing algorithm that tries very hard to optimize on a loss function yet fails to induce generalized alignment.
I could go on but hopefully we can see that this is a major intuition pump.
It's a bad pump and Evolution is a bad analogy for AGI: inner alignment. Enter Quintin Pope, 13th Aug 2022.
One way people motivate extreme levels of concern about inner misalignment is to reference the fact that evolution failed to align humans to the objective of maximizing inclusive genetic fitness. … Evolution didn't directly optimize over our values. It optimized over our learning process and reward circuitry.
The relationship we want to make inferences about is: - "a particular AI's learning process + reward function + training environment -> the AI's learned values"
I think that "AI learning -> AI values" is much more similar to "human learning -> human values" than it is to "evolution -> human values". Steve Byrnes makes this case in much more detail in his post on the matter [23rd Mar 2021].
Evolution is a bi-level optimization process, with evolution optimizing over genes, and the genes specifying the human learning process, which then optimizes over human cognition. … SGD directly optimizes over an AI’s cognition, just as human within-lifetime learning directly optimizes over human cognition.
Or putting this in the «sharp left turn» frame:
within-lifetime learning happens much, much faster than evolution. Even if we conservatively say that brains do two updates per second, and that a generation is just 20 years long, that means a single person’s brain will perform ~1.2 billion updates per generation. … We don't train AIs via an outer optimizer over possible inner learning processes, where each inner learning process is initialized from scratch, then takes billions of inner learning steps before the outer optimization process takes one step, and then is deleted after the outer optimizer's single step. Such a bi-level training process would necessarily experience a sharp left turn once each inner learner became capable of building off the progress made by the previous inner learner. … However, this sharp left turn does not occur because the inner learning processes suddenly become much better / more foomy / more general in a handful of outer optimization steps.… In my frame, we've already figured out and applied the sharp left turn to our AI systems, in that we don't waste our compute on massive amounts of incredibly inefficient neural architecture search, hyperparameter tuning, or meta optimization.
Put another way: it is crucial that SGD optimizes policies themselves, and with smooth, high-density feedback from their performance on the objective function, while evolution random-walks over architectures and inductive biases of policies. An individual model is vastly more analogous to an individual human than to an evolving species, no matter on how many podcasts Yud says «hill climbing». Evolution in principle cannot be trusted to create policies that work robustly out of distribution: it can only search for local basins of optimality that are conditional on the distribution, outside of which adaptive behavior predicated on stupid evolved inductive biases does not get learned. This consideration makes the analogy based on both algorithms being «hill-climbing» deceptive, and regularized SGD inherently a stronger paradigm for OOD alignment.
But Yud keeps making it. When Quintin wrote a damning list of objections to Yud's position (using Bankless episode as a starting point), a month ago, he brought it up in more detail:
This is an argument [Yud] makes quite often, here and elsewhere, and I think it's completely wrong. I think that analogies to evolution tell us roughly nothing about the difficulty of alignment in machine learning.
… Moreover, robust alignment to IGF requires that you even have a concept of IGF in the first place. Ancestral humans never developed such a concept, so it was never useful for evolution to select for reward circuitry that would cause humans to form values around the IGF concept.
[Gradient descent] is different in that it directly optimizes over values / cognition, and that AIs will presumably have a conception of human values during training.
[Ice cream example] also illustrates the importance of thinking mechanistically, and not allegorically.
the reason humans like ice cream is because evolution created a learning process with hard-coded circuitry that assigns high rewards for eating foods like ice cream.
What does this mean for alignment? How do we prevent AIs from behaving badly as a result of a similar "misgeneralization"? What alignment insights does the fleshed-out mechanistic story of humans coming to like ice cream provide?
As far as I can tell, the answer is: don't reward your AIs for taking bad actions.
That's all it would take, because the mechanistic story above requires a specific step where the human eats ice cream and activates their reward circuits.
Compare, Yud'07: «Cognitive causes are ontologically distinct from evolutionary causes. They are made out of a different kind of stuff. Cognitive causes are made of neurons. Evolutionary causes are made of ancestors.» And «DNA constructs protein brains with reward signals that have a long-distance correlation to reproductive fitness, but a short-distance correlation to organism behavior… We, the handiwork of evolution, are as alien to evolution as our Maker is alien to us.»
So how did Yud'23 respond?
This is kinda long. If I had time to engage with one part of this as a sample of whether it holds up to a counterresponse, what would be the strongest foot you could put forward?
Then he was pitched the evolution problem, and curtly answered the most trivial issue he could instead. «And that's it, I guess».
So the distinction of (what we in this DL era can understand as) learning policies and evolving inductive biases was recognized by Yud as early as in 2007; the concrete published-on-Lesswrong explanation why evolution is a bad analogy for AI training dates to 2021 at the latest; Quintin's analysis is 8+ months old; this hasn't had much effect on Yud's rhetoric about evolution being an important precedent supporting his pessimism, nor on the conviction of believers that his reasoning is sound.
It seems he's just anchored to the point, and strongly feels these issues are all nitpicks, and the argument should still work, one way or another, at least it proves that something-kinda-like-that is likely and therefore doom is still inevitable – even if evolution «does not use calculus», even if the category of «hill-climbing algorithms» is not informative. He barely glanced at what gradient descent does, and concluded that it's an optimization process, thus he's totally right.
People who try sniffing "nobody in alignment understands real AI engineering"... must have never worked in real AI engineering, to have no idea how few of the details matter to the macro arguments. … Or, of course, if they're real AI engineers themselves and do know all those technical details that are obviously not relevant - why, they must be lying, or self-deceiving so strongly that it amounts to other-deception, when they try that particular gambit for bullying and authority-assertion.
1/2
You can explain to me how this new and exciting theory of the universe, that hinges entirely on mathematical assumptions, is like dumping a gallon of milk into a box of cereal before pouring it into the bowl, and I can maybe relate to that analogy because I know milk and cereal. But, again, at the end of the day I will never be able to relate that analogy to what is actually being talked about because all that's really there is theoretical math I don't understand.
Your broad impression is correct with one massive caveat: there's no there, there. It is about milk and cereal, and the pretense that the analogy simplifies some profound idea is a pose, it serves to belittle and bully you into meek acceptance of the conclusion which is not founded on some solid model applying to the bowl and the universe alike. Yud's opinions do not follow from math, he arrived at them before stumbling on convenient math, most other doomers don't even understand involved math, and none of this math says much of anything about AI we are likely to build.
It's important to realize, I think, that Yud's education is 75% Science Fiction from his dad's library and 25% Jewish lore in cheder he flunked out of. That's all he learned systematically in his life, I'm afraid; other than that he just skimmed Kahnemahn, Cialdini and so on, assorted pop-sci, and some math and physics and comp sci because he is, after all, pretty smart and inclined to play around with cute abstractions. But that's it. He never had to meet deadlines, he never worked empirically, he never applied any of the math he learned in a way that was regularized against some real-world benchmark, KPI or a mean professor. Bluntly, he's a fraud, a simulacrum, an impostor.
More charitably, he's a 43-year-old professional wunderkind whose self-perception hinges on continuing to play the part. He's similar to Yevgeny «Genius» «Maestro» Ponasenkov, a weird fat guy who LARPs as a pre-Revolutionary noble and a maverick historian (based). Colloquially these people are known as freaks and crackpots, and their best defense for the last two millenia is that Socrates was probably the same but he became Great; except he did not LARP as anyone else.
I know this dirty observation is not polite to make among Rationalists. I've talked to really smart and accomplished people who roll their eyes when I say this about Yud, who object «come on now, you're clowning yourself, the guy's some savant – hell, I've got a Ph.D in particle physics and won at the All-Russian Math Olympiad, and he's nobody but talks jargon like he understands it better than my peers» and I want to scream «you dumb defenseless quokka, do you realize that while you were grinding for that Olympiad he was grinding to give off signals of an epic awesome Sci-Fi character?! That for every bit of knowledge, he gets a hundredfold more credit than you, because he arranges it into a mask while you add to the pearl of your inner understanding? That the way Yud comes across is not a glimpse of his formidability but the whole of it? Can you not learn that we wordcels are born with dark magic at the tips of our tongues, magic you do not possess, magic that cannot remake nature but enslaves minds?»
Ahem.
Let's talk about one such analogy, actually the core analogy he uses: it's about human evolution and inclusive genetic fitness. AGI Ruin: A List of Lethalities, 5th Jun '22:
Section B:
So why not train a giant stack of transformer layers on a dataset of agents doing nice things and not bad things, throw in the word 'corrigibility' somewhere, crank up that computing power, and get out an aligned AGI?
Section B.2: Central difficulties of outer and inner alignment.
16. Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again: the first semi-outer-aligned solutions found, in the search ordering of a real-world bounded optimization process, are not inner-aligned solutions. This is sufficient on its own, even ignoring many other items on this list, to trash entire categories of naive alignment proposals which assume that if you optimize a bunch on a loss function calculated using some simple concept, you get perfect inner alignment on that concept.
Point 16, Misalignment In The Only Precedent We Know About, is a big deal. There are 46 points in total, but it's a bit of a sham: many about AGI being smart, politics of «preventing other people from building an unaligned AGI», handwringing in 39-43, «multiple unaligned AGIs still bad», other padding. Pretty much every moving part depends on the core argument for AI being very likely to «learn wrong» i.e. acquire traits that unfold as hazardous out of (training) distribution, and the 16th corroborates all of such distributional reasoning in B.1 (10-15). 17-19, arguably more, expound on 16.
Accordingly, Yudkowsky cites it a lot and in slightly varied forms, e.g. on Bankless, 20th Feb 23:
we do not know how to get goals into a system. We can cause them to do a thing inside a distribution they were optimized over using gradient descent. But if you shift them outside of that distribution, I expect other weird things start happening. … GPT-7, there's probably a bunch of stuff in there too that desires to accurately model things like humans under a wide range of circumstances, but it's not exactly humans because ice cream didn't exist in the natural environment, the ancestral environment, the environment of evolutionary adaptedness. There was nothing with that much sugar, salt, fat combined together as ice cream. We are not built to want ice cream. We were built to want strawberries, honey, a gazelle that you killed and cooked … but then ice cream comes along and it fits those taste buds better than anything that existed in the environment that we were optimized over.
You can nonetheless imagine that there is this hill climbing process, not like gradient descent, because gradient descent uses calculus, this is just using like, where are you? But still, hill climbing in both cases makes things something better and better over time, in steps, and natural selection was optimizing exclusively for this very simple, pure criterion of inclusive genetic fitness in a very complicated environment. We're doing a very wide range of things and solving a wide range of problems led to having more kids, and this got you humans which had no internal notion of inclusive genetic fitness until thousands of years later, when they were actually figuring out what had even happened, and no desire to, no explicit desire to increase inclusive genetic fitness. So from this important case study, we may infer the important fact that if you do a whole bunch of hill climbing on a very simple loss function, at the point where the system's capabilities start to generalize very widely, when it is in an intuitive sense becoming very capable and generalizing far outside the training distribution, we know that there is no general law saying that the system even internally represents, let alone tries to optimize the very simple loss function you are training it on.
(Distinguishing SGD from an evolutionary algorithm with the mention of «calculus» is a bit odd).
And on Twitter, April 24th 2023 :
…for example, hominid evolution falsifies any purported general law along the lines of "hill-climbing optimization for a loss function, to the point where that produces general intelligence, produces robust generalization of the intuitive 'meaning' of the loss function even as the system optimized becomes more intelligent". Humans were optimized purely for inclusive genetic fitness, and we ended up with no built-in internal psychological concept of what that is. When we got smarter, smart enough that condoms were a new option that didn't exist in the ancestral environment / training distribution, we started using condoms. Gradient descent isn't natural selection, but…
It's not just Yudkowsky these days but e.g. Evan Hubinger, AI safety research scientist at Anthropic, the premier alignment-concerned lab, in 2020.
And Yud's Youtube evangelist Rob Miles, Apr 21, 2023:
@ESYudkowsky I propose this as a clearer example to support "Humans are not trying to maximise inclusive genetic fitness even a little bit"
It definitely is the ultimate cause of our motivations, emotions, and values, my point is just that this fact is not sufficient for us to explicitly try to get it
DaseindustriesLtd
late version of a small language model
3yr ago
Thank you for explaining your perspective concretely, this clears things up.
DaseindustriesLtd
late version of a small language model
3yr ago
I admit this is surprising. I would've predicted the Butlerian Jihad movement deprioritizing Yud as a crank who may blurt out some risky political take, but he establishes himself more and more as the Rightful Caliph. Have Yuddites discovered a stash of SBF's lunch money to buy a bunch of podcasters, including some crypto has-beens looking for a new grift? Or is this simply a snowball effect, where Yud is becoming more credible and attractive the more he goes to podcasts?
On the other hand, this is all show for the plebs anyway; Policy people never lack for experts to cite. And «rationalists» can straight up lie to their audiences even about words of those experts.
I should accelerate my work on a dunk on Yudkowsky's whole paradigm, even though it honestly feels hopeless and pointless. If anyone has better ideas, I'm all ears.
DaseindustriesLtd
late version of a small language model
3yr ago
I do think most of you harbor a lot of racial ill-will, and you use HBD to justify it.
In a nutshell, resentment and animosity, or lack thereof.
Thanks, this is sufficiently clear. It could only be better if you named names, instead of spreading this accusation thin on a group of people many of whom, I believe, do not materially differ from you in either opinion or feeling about race.
What "treatment" has anyone received from me, other than disagreement and (for the overt racialists) scorn?
Mealy-mouthed but persistent gaslighting about the nature of one's feelings and motivations with regard to the subject.
You can start making memes.
The Chinese will gladly trade the vanishing shot at dominance for some more overregulation and domestic oppression in the short term. Indeed, that's how they settled the issue of Choynese Eugenics, and this is what they're doing already with AI. That's how that nation rolls, that's their part in the dance, being a role model in all things illiberal for creepy globalist technocrats, and simultaneously a feeble, unappealing caricature that Americans won't feel too guilty erasing. Putin, too, could've been building GPU clusters Yud suggests must be bombed. Instead he preemptively bombed Mariupol. There is no «geopolitics», it's all a sham.
Not much more.
The rhetoric of schizos Leahy and Yud is on every outlet, from niche forums to highbrow magazines to popular podcasts to mainstream media to the White House and the omnipotent Chuck now. Yud must be imagining this is an anthropic shadow – since he's so smart and policymakers are stupid and cannot coordinate, it can't be that his premises are revealed as wrong now that they endorse his message, it can't be that he's a patsy, it must be some issue of broken multiversal causality! If a grown man thinks he's an anime character, one needn't deceive him any more on top of that.
Importantly they face no credible opposition – LeCun spews condescending inarticulate nonsense and doesn't call Yud out on his subtle technical misstatements; and the non-technical folks just accept the Terminator imagery because why not, it's intuitive! The Overton window is being so defined as to make their conclusion the obviously accepted one, devoid of serious opposition. Even alignment doomers like Christiano who don't live in the la-la land are sidelined.
Once more, a chance at escaping the predefined trajectory that was dangled before my eyes is being pulled away. Crypto – regulated, FUDed into the ground and drowned in speculative nonsense, many founders dead. Darknet and darkmarket – busted. Citizen science, biohacking – fizzled out, and good luck after COVID. AI could have been our shot at greater autonomy, it could've helped everyone, from the unhinged transhumanist to the paranoid traditionalist, divest from the failing body politics and global supply chains. But no, that's too rich.
The powers that be prefer preservation of the status quo more than they yearn for new capability; they've ruined several technologies with the precautionary principle already. And they'll have any capability they want if they ensure strong AIs do not leak outside the regulated state-mandated bounds.
My main bet now is the Wall Street, big tech, and their surprisingly moderate pocket regulators who throw minor bones to the politicians – but China hoped on that mercantile lobbyism, and look where this got us and them.
As usual, hope to be just overemphasizing the bad scenario.
More options
Context Copy link