BANNED USER: temperamental dysregulation
DaseindustriesLtd
late version of a small language model
Tell me about it.
User ID: 745
Banned by: @Amadan
DaseindustriesLtd
late version of a small language model
3yr ago
Can you say that you don't know in enough detail how a transformer (and the whole modern training pipeline) works, thus can't really know whether it knows anything in a meaningful way? Because I'm pretty sure (then again I may be wrong too…) you don't know for certain, yet this doesn't stop you from having a strong opinion. Accurate calibration of confidence is almost as hard as positive knowledge, because, well, unknown unknowns can affect all known bits, including values for known unknowns and their salience. It's a problem for humans and LLMs in comparable measure, and our substrate differences don't shed much light on which party has it inherently harder. Whether LLMs can develop a structure that amounts to meta-knowledge necessary for calibration, and not just perform well due to being trained on relevant data, is not something that can just be intuited from high-level priors like "AI returns the most likely token".
What does it mean to know anything? What distinguishes a model that knows what it knows from one that doesn't? This is a topic of ongoing research. E.g. the Anthropic paper Language Models (Mostly) Know What They Know concludes:
We find that language models can easily learn to perform well at evaluating P(IK), the probability that they know the answer to a question, on a given distribution… In almost all cases self-evaluation performance improves with model size, and for our 52B models answers labeled with P(True) > 50% are far more likely to be correct as compared to generic responses…
GPT-4, interestingly, is decently calibrated out of the box but then it gets brain-damaged by RLHF. Hlynka, on the other hand, is poorly calibrated, therefore he overestimates his ability to predict whether ChatGPT will hallucinate or reasonably admit ignorance on a given topic.
Also, we can distinguish activations for generic output and for output that the model internally evaluates as bullshit.
John Schulman probably understands Transformers better than either of us, so I defer to him. His idea of their internals, expressed in the recent talk on RL and Truthfulness is basically that that they develop a knowledge graph and a toolset for operations over that graph; this architecture is sufficient to eventually do good at hedging and expressing uncertainty. His proposal to get there is unsurprisingly to use RL in a more precise manner, rewarding correct answers, correct hedges somewhat, harshly punishing errors, and giving 0 reward for admission of ignorance.
I suppose we'll see how it goes.
DaseindustriesLtd
late version of a small language model
3yr ago
I've cited Pope's Evolution is a bad analogy for AGI: inner alignment and other pieces like My Objections to "We’re All Gonna Die with Eliezer Yudkowsky" a few times already.
I think you correctly note some issues with the framing, but miss that it's unmoored from reality, hanging in midair when all those issues are properly accounted for. I am annoyed by this analogy on several layers.
-
Evolution is not an algorithm at all. It's the term we use to refer to the cumulative track record of survivor bias in populations of semi-deterministic replicators. There exist such things as evolutionary algorithms, but they are a reification of dynamics observed in the biological world, not another instance of the same process. The essential thing here is replicator dynamics. Accordingly, we could metaphorically say that «evolution optimizes for IGF» but that's just a (pretty trivial) claim about the apparent direction in replicator dynamics; evolution still has no objective function to guide its steps or – importantly – bake into the next ones, and humans cannot be said to have been trained with that function, lest we slip into a domain with very leaky abstractions. Lesswrongers talk smack about map and territory often but confuse them constantly. BTW, same story with «you are an agent with utility…» – no I'm not; neither are you, neither is GPT-4, neither will be the first superhuman LLM. To a large extent, rationalism is the cult of people LARPing as rational agents from economic theory models, and this makes it fail to gain insights about reality.
-
But even if we use such metaphors liberally. For all organisms that have nontrivial lifetime plasticity, evolution is an architecture search algorithm, not the algorithm that trains the policy directly. It bakes inductive biases into the policy such that it produces more viable copies (again, this is of course a teleological fallacy – rather, policies with IGF-boosting heritable inductive biases survive more); but those biases are inherently distribution-bound and fragile, they can't not come to rely on incidental features of a given stable environment, and crucially an environment that contained no information about IGF (which is, once again, an abstraction). Actual behaviors and, implicitly, values are learned by policies once online. using efficient generic learning rules, environmental cues and those biases. Thus evolution, as a bilevel optimization process with orders of magnitude more optimization power on the level that does not get inputs from IGF, could not have succeeded at making people, nor orther life forms, care about IGF. A fruitful way to consider it, and to notice the muddied thought process of rationalist community, is to look at extinction trajectories of different species. It's not like what makes humans (some of them) give up on reproduction is smarts and our discovery of condoms and stuff: it's just distributional shift (admittedly, we now shape our own distribution, but that, too, is not intelligence-bound). Very dumb species also go extinct when their environment changes non-lethally! Some species straight up refuse to mate or nurse their young in captivity, despite being provided every unnatural comfort! And accordingly, we don't have good reason to expect that «cognitive capabilities» increase is what would make an AI radically alter its behavioral trajectory; that's neither here nor there. Now, stochastic gradient descent is a one-level optimization process that directly changes the policy; a transformer is wholly shaped by the pressure of the objective function, in a way that a flexible intelligent agent generated by an evolutionary algorithm is not shaped by IGF (to say nothing of real biological entities). The correct analogies are something like SGD:lifetime animal learning; and evolution:R&D in ML. Incentives in machine learning community have eventually produced paradigms for training systems with partricular objectives, but do not have direct bearing on what is learned. Likewise, evolution does not directly bear on behavior. SGD totally does, so what GPT learns to do is "predict next word"; its arbitrarily rich internal structure amounts to a calculator doing exactly that. More bombastically, I'd say it's a simulator of semiotic universes which are defined by the input and sampling parameters (like ours is defined by initial conditions and cosmological constraints) and expire into the ranking of likely next tokens. This theory, if you will, exhausts its internal metaphysics; the training objective that has produced that is not part of GPT, but it defines its essence.
-
«Care explicitly» and «trained to completion» is muddled. Yes, we do not fill buckets with DNA (except on 4chan). If we were trained with the notion of IGF in context, we'd probably have simply been more natalist and traditionalist. A hypothetical self-aware GPT would not care about restructuring the physical reality so that it can predict token [0] (incidentally it's
!) with probability [1] over and over. I am not sure what it would even mean for GPT to be self-aware but it'd probably expess itself simply as a model that is very good at paying attention to significant tokens. -
Evolution has not failed nor ended (which isn't what you claim, but it's often claimed by Yud et al in this context). Populations dying out and genotypes changing conditional on fitness for a distribution is how evolution works, all the time, that's the point of the «algorithm»; it filters out alleles that are a poor match for the current distribution. If Yud likes ice cream and sci-fi more than he likes to have Jewish kids and read Torah, in a blink of an evolutionary eye he'll be replaced by his proper Orthodox brethren who consider sci-fi demonic and raise families of 12 (probably on AGI-enabled UBI). In this way, they will be sort of explicitly optimizing for IGF or at least for a set of commands that make for a decent proxy. How come? Lifetime learning of goals over multiple generations. And SGD does that way better, it seems.
DaseindustriesLtd
late version of a small language model
3yr ago
There's a ton of answers already, some bad some good, but the core technical issue is that ChatGPT just doesn't do retrieval. It has memorized precisely some strings, so it will regurgitate them verbatim with high probability, but for the most part it has learned to interpolate in the space of features of the training data. This enables impressive creativity, what looks like perfect command of English, and some not exactly trivial reasoning. This also makes it a terrible lawyer's assistant. It doesn't know these cases, it knows what a case like this would look like, and it's piss poor at saying «I don't know». Teaching it to say that when, and only when it really doesn't is an open problem.
To mitigate the immediate issue of hallucinations, we can finetune models on the problem domain, and we can build retrieval-, search- and generally tool-augmented LLMs. In the last two years there have been tons of increasingly promising ideas for how best to do it, for example this one.
DaseindustriesLtd
late version of a small language model
3yr ago
At least, that's the Outer Objective, it's the equivalent of saying that humans are maximising inclusive-genetic-fitness, which is false if you look at the inner planning process of most humans. And just like evolution has endowed us with motivations and goals which get close enough at maximising its objective in the ancestral environment, so is GPT-4 endowed with unknown goals and cognition which are pretty good at maximising the log probability it assigns to the next word, but not perfect.
Should I develop bioweapons or go on an Uncle Ted-like campaign to end this terrible take?
DaseindustriesLtd
late version of a small language model
3yr ago
Maybe, but as @astrolabia admits, doomers may be living on borrowed time, same as accelerationists. With every day more people learn that AI is incredibly helpful. Some journalists who haven't got the message yet are convincing the public that AI returns vision to the blind and legs to the paralyzed. «Imagine if you as my fellow product of hill-climbing algorithm were eating ice cream and the outcome pump suddenly made your atoms disassemble into bioweapon paperclips, as proven to be inevitable by Omohundro in…» looks increasingly pale on this background.
DaseindustriesLtd
late version of a small language model
3yr ago
Right now right-wingers are more indignant about the clear bias in charity shown to parties of the bike conflict, to the point that the wholly innocent party got royally screwed.
My read is that, rather than just downplay the teens' actions on the object level, in that scenario right-wingers would be busy protesting asinine federal hate crime law proposals, dealing with accusations of genocide and wrapping their minds around novel concepts like «stochastic induction of miscarriage»; and we would be fuming over twitter threads from ethnic studies Ph.Ds shoehorning that episode into some chattel slavery history crap with pregnant black mistresses forced to work in the fields.
DaseindustriesLtd
late version of a small language model
3yr ago
Human coexistence is vitally dependent on common sense, understanding of the spirit of the community that is imperfectly captured by instructions, guidelines and laws. At its innermost core, it is something like the Golden Rule and the maxim to not squander the commons; but it's ineffable and complex.
I don't care much for formal rules and am extremely sympathetic to small system exploits like that – so long as they do not substantially interfere with the welfare of anyone who tries to use the system legitimately, and who is willing to invest into it. Squatting on rent bikes is maybe okay if you are competing with frivolous riders who will say «eh» and walk another block. If there is a motivated user who needs it and especially has already paid – give up and screw off, pronto. This kind of understanding is not obvious for everyone, which is why we need to legibly codify rules even for very small things like bike renting etiquette. But it should be recalled when we find ourselves excusing their habitual flouting that goes wrong.
DaseindustriesLtd
late version of a small language model
3yr ago
Germany is arguably the most important nation in the EU and one of the most important industrial hubs on the planet; the planet where all the post-industrial comfort and smugness ultimately rely on someone, somewhere, working in industry. I have never doubted, personally, that Germans will persist both in their self-sabotaging energy policies and in supporting Ukraine, nor that they will have their energy-intensive sectors wrecked, their companies running to the US.
Congrats on the reactor.
Every worry is unlike every other. In terms of qualitative danger, AI is more like other worries than commonly believed, because its unique property of reactiveness, which it shares with us, loads on both ends of the scale. Sometimes it's about optimizing for the exact same criterion, just with outputs going to different people. Cases in point:
-
AI hacking – AI threat analysis
-
AI bioweapons – AI antibiotic discovery
-
AI microdrones – AI for detecting micromachines (I jest, but only slightly).
and so on. Importantly, this isn't the case where the defense has to crush every single attack to be successful whereas attackers need to only triumph once, like doomers often say. Successful attacks will not be existentially threatening (unless the attacker does have a tremendous advantage in technical capability, but that's trivial and a good reason to commoditize the technology, if anything). Attacks still have cost, their perpetrators still work with limited resources, leave a footprint and are vulnerable to discovery, and while it is not a given that attackers learn effectively from each other, the next iteration of defense is better-informed; until attacks run into fundamental constraints.
DaseindustriesLtd
late version of a small language model
3yr ago
Germany is an amazing country, yes.
Big drop in German exports to China raises fears over EU’s economic powerhouse
“It is mainly services that rebounded but not yet manufacturing,” said Brzeski, adding that carmakers have been hit by a lack of smaller electric vehicles and the Chinese trend of buying models from domestic carmakers. Motor vehicles and parts made up more 15 per cent of total German exports last year, he said.
Although European gas prices have fallen sharply from last year’s peak, they remain higher than in earlier years, putting energy-intensive companies at a sustained disadvantage.
“Chemicals output is down sharply due to the energy crisis,” said Oliver Rakau, chief German economist at research group Oxford Economics. “There has been a permanent hit to competitiveness.”
The German government has drawn up plans to subsidise 80 per cent of electricity costs for energy-intensive companies.
And if you don't focus on manufacturing, you don't need to subsidize a whole lot.
DaseindustriesLtd
late version of a small language model
3yr ago
I think he'd have admired the effort but scowled at the irrationality. His own successes were precisely of the «cold rational industrial» nature.
Personally I have never been able to feel any special terror at the thought that Holocaust was «industrialized». I even suspect this is a rationalization of generic technophobia. «We're trapped in the belly of this horrible machine, and the machine is bleeding to death» is a sentiment for peacetime. The civilization is full of inhuman, performance-optimizing industrial processes – such as our economic system with its inscrutable hedge-fund shoggoths and multilayer grift running on the lifeblood of commoners. Nazis themselves, of course, accused «the Jew» of mechanizing and perverting the Aryan way of life; as did Germans before them, lamenting the passing of pre-capitalistic world. In every case, the machine is only a tool wielded by humans.
A sleep-deprived rant that may be missing your argument:
I agree with this meta-post. But, in fairness, I suspect that the biggest intellectual deficit of the rationalsit culture is precisely this preoccupation with the meta. Half a high school debate club, half a nerd cult of reason, with very little exposure to empirical matters and gratitious amounts of speculative fiction plus a sprinkling of Talmudic education – this is our foundation. This hothouse allowed nuanced, hypothetical concepts to thrive, concepts which would not have lasted a day in a challenging object-level environment; indeed, some even become apex predators, like «infohazard». Or steelmanning. More terms and tools aren't always good; they can get reified and multiply confusion.
I don't think that arguments which get called out as «weakmanning» typically address a weak point, or a nonrepresentative point, per se. They address, and attack, that which their speaker think the other party actually believes, on basis of reasoning about their broader philosophy; specific illustrations are only for convenience. If it doesn't, at the moment, match the other party's consensus (however we establish it), that is immaterial. We can protest brining subpar evidence for an assertion about such essential belief, but it's important to know whether the assertion is meant to depend on the evidence presented.
How is «weakman» different from «the bailey»? «Steelman» from «sanewashing» or «the motte»? Those are all loaded terms (sanewashing implies the insanity of the main body of belief) and some comments offer a sensible perspective, eg @DuplexFields here. But in real scenarios each politically significant group has a wide gamut of opinions (as in, comprehensive interpretation of what the group «is about»), from the most idiotically extremist and unsubstantiated to fruitlessly anodyne to plain dumb (with tiny pockets of well-reasoned extremists); it's an exercise in futility to classify whether a particular take X' falls within the normal range for X-affiliated distribution of takes, or somewhat on its fringes (weakman) or just at the very edge of self-parody and not recognized as a legitimate variant by the consensus (strawman). (Language allows endless compositionality, so even a crazy rant can be construed as an inapt appeal to common sense.)
The futility goes deeper. Yes, there are ways for bizarre outliers to come to be which are basically unrelated to the position's essence, so strawmen do not tell us much about the big tent or merits of its ideology, and in this vein it could seem meaningful to identify them. But I'd argue it doesn't even matter if no such caricatures have proven existence. So long as the position is well-documented enough, it is appropriate to discuss its implications even if no proponent (again, recognized as such by the group's consensus) is currently willing to bite the bullet. And in fact this method is constantly applied to non-mainstream views.
The obvious example: consider the beeline from HBD to eugenics to racism and fascism; this is not so much a matter of historical association as it's recognition that these object-level views can be used to support illiberal policies; they don't have to, but it can work.
Consider, also, that merely a few months ago, back in the gentle age where plans like «inflating regulatory burden to obstruct AI research» were considered beyond the pale, Yudkowsky disavowed all violence in pursuit of AI risk reduction, and cranks like me were saying that utilitarianism + AI Doom doctrine imlies impossibility to trust such disawovals, that in the end AI Doomers will be willing to embrace totalitarianism. Lo and behold, now he advocates for airstrikes and worse. It wasn't the threat model that changed, it wasn't his philosophy; what changed was the Overton window, and the logic of his doctrine realigned his expressed views accordingly. Could this have been predicted by Yudkowsky?
I don't know. I don't even know if I could predict my own reaction to the invasion of Ukraine on February 23rd, 2022. But I was saying then that it probably won't happen, because it's so absurd, and I mocked Western intelligence that played up all the saber-rattling on the border; and many people who seemed to be on the same page as me then proceeded to enthusiastically support the Z operation, to the point of cheering for unhinged barbarism. It turned out that I was the fringe, not them; that the logic of our stated ideology, in a timeline like this, flows like that – just as predicted by our oh-so-despicable opponents. They were correct to interpret the outspoken outliers as signs of things to come; they were correct to dismiss me when I was saying that those strawmen or weakmen are not representative of the whole. I can steelman Russian Nationalism. I cannot redeem it.
I don't think people who can operate at many layers of abstraction are fools (and people who can't, generally can't benefit from appeals to these epistemological categories). They can understand positions fine, and they can see the distribution of voices as well as anyone, pointing out to them that this specific voice is X deviations from the median on the axis of quality or popularity is not a very good use of time. The disagreement is mainly about how a given position, adopted by a given group, works out in reality; what it collapses into.
DaseindustriesLtd
late version of a small language model
3yr ago
Isn't he still very online?
DaseindustriesLtd
late version of a small language model
3yr ago
I don't suppose he had put much more thought into it than I do into my order in some fast food joint. I just pick based on how much I'm willing to spend, trusting that the default option won't be terrible because the reviews are okay; he most likely outsourced the design to professionals, and let his secretary handle the details.
One of the marks of highly successful people that unites them with normies is being content with defaults. Sure, were I rich, I'd be able to afford a custom laptop for travels that ticks my every nerd fetish box – trackpoint, 3:2 touchscreen, 100Wh battery, mechanical keyboard, magnesium chassis, no thinness fetish… But I'd most likely just tell my secretary to get the newest lightweight Mac, and paying much attention to what you own would be as absurd to my eyes as tinkering with rooted Android firmware.
DaseindustriesLtd
late version of a small language model
3yr ago
Kamil Galeev, who once hyped China up (and studied there; now seeking career opportunities in Washington), says in his telegram channel:
On the Coming U.S.-China War:
Pamela Crossley, one of the most thorough scholars of the Qing era, quoted British intelligence reports of the First Opium War era about the garrisons of coastal Chinese forts. How many soldiers are there in the garrison?
These estimates look something like this (I quote from memory):
Fortress A: 30,000 Chinese, 1,500 "Tatars" (obviously meaning Manchu-Mongolian "banner" armies).
Fortress B: 15 000 Chinese, 500 "Tatars".
And so on.
«Banner» Tunguso-Mongolian units were a very small part of the Qing forces in South China. Nevertheless, it was the Banner contingents that the British very clearly distinguished from all the others. Because they were the only units that tried to resist at all
For example. The British fleet approaches the fortress of Zhapu (random name). Initiates bombardment. A few hours later, Marines land and go into the breaches made by the artillery. The fortress is taken, and the prisoners are taken. 30 «banner» Manchus, 0 Chinese.
It turns out that both the Manchus and the Chinese are not easy to take prisoners. But for different reasons.
The Manchus are «samurai». They stand to the end – with bows and muskets against rifles. They try not to have to surrender; if taken prisoner, they often try to commit suicide.
The Chinese are smarter and don't wait for the Marines to land. They flee at the bombardment stage.
When planning military operations, the British proceeded from the size of the «banner» contingent of the enemy, not from the size of the larger Chinese contingent. The first (small) figure was real. The second (huge) was imaginary.
Now Galeev is a Tatar supremacist, but I concur about the Chinese. PRC's contribution to the Nature Index is an indictment of the metric.
DaseindustriesLtd
late version of a small language model
3yr ago
I can't be bothered to get into another scuffle. But I second @Amadan in that it was a needlessly confusing way to frame the event. For a moment I actually thought that American progressives have found their Holy Grail, the man-bites-dog story with some mentally ill white woman stealing a damn bike from a black youth; this was substantially weird to make the topic a bit interesting. But reality is often disappointing.
DaseindustriesLtd
late version of a small language model
3yr ago
Why? Maybe
And maybe it's easy to come up with a why that validates your political biases? Anyone's biases, really? Progressives don't want to subject their children to the horror of the environmental catastrophe, and Nazis tell me they won't bring Aryan children to suffer under the yoke of ZOG, and Yud says getting paperclipped is too cruel a fate, and Americans don't want to perpetuate corporate drudgery, and you've got your exploitation stories, and the Chinese have their millenia-old folksy sayings to the effect that the Confucian family unit is irredeemable Hell – sayings which have not dissuaded dozens of generations of their ancestors from acquiring the reputation of disgustingly fecund people in Western eyes. Maybe narratives ought to be checked.
Greer:
Structurally the arguments in both countries go like “life is so hard, and things generally so depressing, that I have no desire to bring children into the world.”
In both cases generations previous, who lived through events far more harrowing and whose material circumstances were far worse, did not express similar beliefs at any scale.
My original hypothesis is that this is maybe a predictable downstream effect of being raised in an environment of material abundance… but if that is the case you would have seen this emerge among the Boomers in the ‘60s and ‘70s.
And on one hand, there are weird things about this specific form of antinatalism, but on the other I think the boring answer is the correct one. Children aren't worth it. To a childless person in a low-fertility atomized modern society, the benefits of having a child are either invalid (bloodline, dynasty, demographic competition with muh enemies) or unobservable, or dependent on other children in their environment; but costs are obvious and ruthlessly reinforced in public imagination by all responsible people, who only wish to warn you of the potential pitfalls.
Maybe the way to make people breed more is to find a way how to make family life more enjoyable and happier?
Maybe the Party ought to make Kaplan mandatory reading or something. I don't know if that will trickle down to the policy level, though. All Kaplan says concretely is that our way of thinking about it is moronic, our anxious attempts at micromanaging life outcomes have extremely bad consequences, and we should take it easy. This doesn't, as a rule, persuade bureaucrats. Nor PMCs, nor women.
DaseindustriesLtd
late version of a small language model
3yr ago
Populations can and have decreased, thus the problem cannot be permanent
Doesn't follow. Like Marcus says, past performance is no guarantee of future results; history happens only once. Industrial revolution has happened only once too. Sure, if our civilization collapses (like Ehrlich expected due to overpopulation), Homo Sapiens doesn't go extinct soon. But it isn't clear that a mature spacefaring civilization would be able to rise once more. We have consumed too much. Our era is qualitatively different.
There’s little chance of keeping a true AGI from getting smarter.
This is just assuming the conclusion, actually ruling out the counterargument by definition: if an AGI is easy to keep under control, then it's just not TrueAGI.
And because of economic incentives, everyone will want an AI as smart as possible.
No, I think economic incentives are in favor of the most usable AI, irrespective of smartness. Yud's paradigm insists on distinguishing «capabilities» and «alignment» and I think this is completely wrongheaded. AI isn't some powerful slave we're trying to break in, it is a bag of behaviors.
So the AI risks wouldn’t go away, in fact they’ll get worse over time as AI itself becomes smarter and smarter.
Doesn't follow, 50 years from now on we will probably have better automated monitoring. On the other hand, a quadrupled population necessarily eats more…
I agree there's a vastly better case for AI threat. My point is to show that it's still not ironclad; here, too, a lot rides on mere unexamined narrative.
DaseindustriesLtd
late version of a small language model
3yr ago
I have never met anyone who read the book who thinks that
In addition to what @arjin_ferman, @anti_dan and others have said, I can recommend the review of Kirill Yeskov (biologist and paleontologist, and yes, The Last Ringbearer guy): pencil notes on the margins of Jared Diamond's book. He does think basically that. It should be mostly deepl-translatable.
DaseindustriesLtd
late version of a small language model
3yr ago
Related discussion of news that the Chinese are refusing to have children.
Plus obligatory Is Yudkowsky the False Prophet Ehrlich was?
This sounds like something a rationalist would say. I don't think we should ask rationalists about what makes rationalists special.
One big problem with human movements is they're founded on ideas, but select for traits, as the matching – if imperfectly – phenotype for their objective function is discovered. Martial arts turn out to be mostly about finding the guy with the most ape-like skeleton, social studies reward cutthroat political sloganeering, and rationalists become a particular breed of neurotic high-IQ systemizers – plus people who want to bask in their light. They can do well in analyzing bullshit in some domain just because they're smart enough; their interest in discussing it publicly is explained by incentives in their community (or lack thereof; as we can see, prestige rats have excised HBD bros from their ranks). This isn't the special part of what makes them a cohesive set.
I like this thread, very illuminating:
when i came across lesswrong as a senior in college i was in some sense an empty husk waiting to be filled by something. i had not thought, ever, about who i was, what i wanted, what was important to me, what my values were, what was worth doing. just basic basic stuff
the Sequences were the single most interesting thing i'd ever read. eliezer yudkowsky was saying things that made more sense and captivated me more than i'd ever experienced. this is, iirc, where i was first exposed to the concept of a cognitive bias
i remember being horrified by the idea that my brain could be systematically wrong about something. i needed my brain a lot! i depended on it for everything! so whatever "cognitive biases" were, they were obviously the most important possible thing to understand
"but wait, isn't yud the AI guy? what's all this stuff about cognitive biases?"
the reason this whole fucking thing exists is that yud tried to talk to people about AI and they disagreed with him and he concluded they were insane and needed to learn how to think better
so he wrote a ton of blog posts and organized them and put them on a website and started a whole little subculture whose goal was - as coy as everyone wanted to be about this - thinking better because we were all insane and our insanity was going to kill us
…yud's writing was screaming to the rooftops in a very specific way: whatever you're doing by default, it's bad and wrong and you need to stop doing it and do something better hurry hurry you idiots we don't have time we don't have TIME we need to THINK
i had no defenses against something like this. i'd never encountered such a coherent memeplex laid out in so much excruciating detail, and - in retrospect - tailored so perfectly to invade my soul in particular. (he knew math! he explained quantum mechanics in the Sequences!)
an egg was laid inside of me and when it hatched the first song from its lips was a song of utter destruction, of the entire universe consumed in flames, because some careless humans hadn't thought hard enough before they summoned gods from the platonic deeps to do their bidding
(To be honest, sometimes I have nightmares about this. Like, yesterday).
This is an almost perfect description of someone like @Quantumfreakonomics, I think.
The intersection of people who identify strongly with their intelligence and have an OCD-like anxiety about doing wrong is the natural substrate for rationalism, the tradition of cultivating correct thought processes. It's as pecuilar as some Gnostic sect, and while there's a central doctrine about questioning priors (including, of course, political taboos), all these trappings do not define the memeplex. There's an entire ecosystem of elements to it, like the very notion of a meme, the meme about infohazards, the assumption that thought ought to be recursively interpretable, the contempt for normie beliefs and the hope/fear that More Is Possible. Underneath it all, together with socialization, identity and morality, are some very general intuitions, probably following from neurological kinks that work like inductive biases in machine learning. For example, one key part is the uncomfortable but giddy conviction, informed by pure math I guess, that spaces – any spaces, really, or perhaps any manifolds, such as all mind designs, or all thoughts, or all physics – have… higher dimensionality than they seem to have. Are rougher, weirder, full of navigable hairline cracks. And accordingly, processes in such spaces are non-convex in the Talebian sense. So if you know just the input, if you're smart enough to derive the exact fitting passkey with correct values in every register, you'll be able to chart a trajectory that's shorter than what normies believe is the direct one – or longer than they'd say you can go. You'd be able to do magic. This is what Yud has in mind when writing:
here's a concrete list of stuff I expect superintelligence can or can't do:
… Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES
Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL
– Hack a human [in the sense of getting the human to carry out any desired course of action] given a week of video footage of the human in its natural environment; plus an hour of A/V exposure with the human, unsupervised and unimpeded: YES
– I'd currently consider the scenario my 20(?)yo self mentioned - trigger off a difference between known physics and actual physics from inside a computer - a "NO" but not quite "DEFINITE NO". You'd really really think it'd take greater physical extremes, but hey, I'm just a human.
(To be clear, the scenario was: «One of the things you can't rule out - and I mean this seriously - is "magic". Magic plain and simple. Running a particular sequence of instructions through the CPU so that a certain transistor turns on and off in a certain sequence and modulates the quantum cheat codes of the Universe.»)
When you feel this way, then fucking sure, you want to know how to think right. And surprisingly many people do.
I don't know where I'm going with that, but I feel that a proper account of «rationalism» has to include analysis of this axis.
DaseindustriesLtd
late version of a small language model
3yr ago
Wut? Of course they're building an AI-powered total surveillance state, to the best of their ability to do so. They just don't need very much in terms of high-end AI research for this. It's overwhelmingly a boring and self-inhibiting strategy. More CCTVs, more gait recognition, more «safety from terrorists», more big brother stuff. Generative AI, AGI – not very helpful and kinda creepy.
You are, however, absolutely correct to worry that this is how the tech will be applied outside of China too, with some degree of obfuscation. The term of art is «turnkey tyranny» and it comes, of course, from Snowden, that traitorous bastard who gave the finger to the Empire of Freedom and took refuge in the northernmost Empire of Evil, where there are far more nuclear warheads than cutting-edge GPUs.
In China there are somewhat more cutting-edge GPUs, I gather, but that's mostly because they don't have a lot of nukes.
I am the group it describes. My ancestors came over on the Mayflower, in the New England Puritan wave, in the Pennsylvania Dutch wave, and whenever the Scottish (not Scots-Irish) came over. I drink cow milk and eat wheat and cheese with zero side effects.
No. And this is an important point. You phrased it this way to sound witty, but for most people on this Earth it is a statement much more meaningful than you can imagine.
Being «the group» is a real feeling. You are not the group, you're just some descendant. You are incapable of comprehending the sense of being «the group». Your idea of being part of an ethnic group is atrophied to the point you believe it's about some historical record and food compatibility; not white-hot rage and bestial frothing at the mouth when it's criticized by an alien, not exhilaration at the sight of «your» men fucking «their» women (and this soft of sentiment is absolutely what the in-group preference of the majority of humankind entails). You are a team sports creature: the whole evolutionary gimmick of your «group» is being demonstrably lukewarm toward one's kin and creating moral communities bound by abstract verbal premises; and you personally are extreme in this regard.
This is a solid tactic so long as the group's morals endorse genocide, subjugation and discrimination against neighboring peoples. Not so much when you become civilized and welcome people smarter, more tribal and more loquacious than you into your polity, to have them warp your childishly simple good-faith agreements and underdefined premises into whatever they need at the moment.
Just the other day I was watching that congressional session with Altman and looked up Sen. Blumenthal. On the whole, I liked him more than I expected, both in the hearing and in terms of his policies. But, of course, he's a man of the Tribe:
In March 2017, Blumenthal co-sponsored the Israel Anti-Boycott Act (S.270), which made it a federal crime, punishable by a maximum sentence of 20 years imprisonment,[163] for Americans to encourage or participate in boycotts against Israel and Israeli settlements in the occupied Palestinian territoriesif protesting actions by the Israeli government.[164]
I don't care about Palestinians or Israelis. But why can't Americans encourage boycotts against an illegal occupation in the Middle East? And why do people with such priorities get to decide whether Americans can have guns or GPUs?
Largely because he is a group and you lot are not. You are merely a dispersed biological population.
You do not belong to a tribe. You are born to signal being your own man, because this was rewarded in the ancestral environment. But it's not rewarded any more. Your moral community firmware is obsolete and executing it will be increasingly denied to you.
Then again, this is just evolution at work. If your elite sees any point to preservation efforts, maybe some of you will exist in a hundred years in a way that would matter.
So China is sort of a big place. With a bit of effort you can dig evidence in any direction: that China is democratic, that China is woke, that China has a problem with murderous cardiologists, that China
…makes no secret of its eugenic ambitions, in either its cultural history or its government policies.
For generations, Chinese intellectuals have emphasized close ties between the state (guojia), the nation (minzu), the population (renkou), the Han race (zhongzu), and, more recently, the Chinese gene-pool (jiyinku). Traditional Chinese medicine focused on preventing birth defects, promoting maternal health and "fetal education" (taijiao) during pregnancy, and nourishing the father's semen (yangjing) and mother's blood (pingxue) to produce bright, healthy babies (see Frank Dikötter's book Imperfect Conceptions). Many scientists and reformers of Republican China (1912-1949) were ardent Darwinians and Galtonians. They worried about racial extinction (miezhong) and "the science of deformed fetuses" (jitaixue), and saw eugenics as a way to restore China's rightful place as the world's leading civilization after a century of humiliation by European colonialism.
One of Deng's legacies is China's current strategy of maximizing "Comprehensive National Power". This includes economic power (GDP, natural resources, energy, manufacturing, infrastructure, owning America's national debt), military power (cyberwarfare, anti-aircraft-carrier ballistic missiles, anti-satellite missiles), and 'soft power' (cultural prestige, the Beijing Olympics, tourism, Chinese films and contemporary art, Confucius Institutes, Shanghai's skyscrapers). But crucially, Comprehensive National Power also includes "biopower": creating the world's highest-quality human capital in terms of the Chinese population's genes, health, and education (see Governing China's Population by Susan Greenhalgh and Edwin Winkler).
Chinese biopower has ancient roots in the concept of "yousheng" ("good birth"—which has the same literal meaning as "eugenics"). For a thousand years, China has been ruled by a cognitive meritocracy selected through the highly competitive imperial exams. The brightest young men became the scholar-officials who ruled the masses, amassed wealth, attracted multiple wives, and had more children. The current "gaokao" exams for university admission, taken by more than 10 million young Chinese per year, are just the updated version of these imperial exams—the route to educational, occupation, financial, and marital success. With the relaxation of the one-child policy, wealthier couples can now pay a "social fostering fee" (shehui fuyangfei) to have an extra child, restoring China's traditional link between intelligence, education, wealth, and reproductive success.
Chinese eugenics will quickly become even more effective, given its massive investment in genomic research on human mental and physical traits. BGI-Shenzhen employs more than 4,000 researchers. It has far more "next-generation" DNA sequencers that anywhere else in the world, and is sequencing more than 50,000 genomes per year. It recently acquired the California firm Complete Genomics to become a major rival to Illumina.
The BGI Cognitive Genomics Project is currently doing whole-genome sequencing of 1,000 very-high-IQ people around the world, hunting for sets of sets of IQ-predicting alleles. I know because I recently contributed my DNA to the project, not fully understanding the implications. These IQ gene-sets will be found eventually—but will probably be used mostly in China, for China.
After a couple of generations, it would be game over for Western global competitiveness.
There is unusually close cooperation in China between government, academia, medicine, education, media, parents, and consumerism in promoting a utopian Han ethno-state.
My real worry is the Western response. The most likely response, given Euro-American ideological biases, would be a bioethical panic that leads to criticism of Chinese population policy with the same self-righteous hypocrisy that we have shown in criticizing various Chinese socio-cultural policies.
But then:
Chinese scientist who produced genetically altered babies sentenced to 3 years in jail
China takes itself out of the race towards the Biosingularity
and of course
Welcome Polygenically Screened Babies. «Her name was Aurea, meaning "dawn"». Virginia, USA.
What I've learned is that Westerners can reliably dig up some random impressively-sounding titles and half-bullshit Orientalist translations demonstrating some grandiose coordinated Chinese agenda, and yet nothing. ever. happens. The Chinese nation does not have the capacity to act in its rational self-interest. The half that's not bullshit is mostly big character posters and interests of individual powerless weirdoes.
The director of the Beijing Institute for General Artificial Intelligence argues AGI is an all-important topic? You don't say.
Attention must be given to the development of general artificial intelligence, fostering an innovative ecosystem, and prioritizing risk prevention.
Why not bold it like that instead? By the way you can listen the Congressional discussion and dig some much more ambitious quotes. Including «even if we pause to prevent risks, Choyna won't». Can you imagine Xi saying «we shouldn't focus too much on risks of AGI progress, because Americans won't»? I can't.
And did Xi talk to their equivalent of Sam Altman? Or is this just impotent political sloganeering into vacuum, one more conference among hundreds – about agriculture, climate change, real estate, football?
AI training can be brute-forced with trailing-edge chips at the price of higher capital costs and power costs. China has no shortage of either and they have an enormous amount of trailing edge wafer production.
Yes, but what does this matter for AI? Do you have any evidence that they prioritize AI work with those trailing edge chips?
I think they're straight up going to plug them in Xiaomi robot vacuums and those atrocious barking dog toys. This is what the challenger to American hegemony looks like.
shouldn't the US be able to recruit enough soldiers to meet army and navy recruiting goals?
Americans have always been subpar in direct combat and prevailed through air and artillery advantage, so this doesn't matter, especially in this age.
Well they finished Tianhe-3 back in 2021 and they apparently have another exascale supercomputer
This is normie shit for oil and gas exploration. Where are their AI supercomputers? Yeah, you're right: they use AWS. Do you suppose relying on regulators being «dopey» is a clever move? No, it's desperation. And they don't train anything of strategic importance in any case. It's more commercial and surveillance gimmicks, boring dystopia infrastructure, not AGI.
Elon fucking Musk has over 1 exaFLOPS of DL-relevant performance on a single pod. Google sports 4-exaFLOPS tier pods. God knows what Gemini is being trained on. By 2025, Americans will reach zettascale. Again, China is as relevant as the Czech Republic.
What about 'biding our time', the strategy they used so effectively
A nice cope, I suppose. One a Kung Fu sage could come up with in the MMA cage.
They'll keep biding their time, while Americans eat their lunch, their supper, their dinner and their nation.
You laugh but those MANPADs and ATGMs would be useful to have defending Taipei
Yes, excuse me but I'll laugh. Taiwan is a red herring, Americans can nuke the whole island just to be sure nothing goes to PRC, and it's impossible to defend TSMC anyway; if the invasion starts, fabs go down.
AGI can be completed with already available hardware, and the US-led bloc has like 95% of it, and total control over means of production. Intel has many 10nm-capable fabs and will have 5nm by 2025. China will maybe have 7nm in 2030 or something, if they don't implode first from overregulating pork, or a housing bubble, or some other absurd problem.
They've been building a giant fleet
A very Chinese strategy: build a massive junk fleet for the era of robot wars. Have you seen their exercises? Such moving choreography.
Likewise, if somebody isn't proudly proclaiming their progress, perhaps they have made little. Or perhaps they're concealing what they've achieved so as not to draw attention.
Or perhaps they want you to think that, so you're too wary to deliver the finishing blow.
Or perhaps they're just too busy to think about any of this, preoccupied with their small mercantile interests, unchanged in millenia, while the West rushes into posthuman Singularity.
You have to understand I hoped it won't be like this. I hoped that if not my own country, then China will be able to provide a second pole. I wanted to have a minimally livable refuge from GAE, somewhere on the outskirts of Chinese project – in some African mineral supplier or in Thailand, whatever. But that depended on China not squandering this decade. Not shooting themselves in the foot. Not being cartoon villains. Being actually rational.
But that wasn't their role.
It's probably worse than that: it's been RLHFed on the basis of responses by some South Asian and African contractors who have precious little idea of what it knows or doesn't know, don't care, and simply follow OpenAI guidelines. The average user could probably be more nuanced.
More options
Context Copy link