This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Or see Bostrom about risks from utilitarian intelligences:
More to the point, consider the name of Yud's Tumblr: Optimize Literally Everything. In Global Risk, he gives the following anodyne definition:
Optimization sounds more humble than maximization of value, but I think they just mean the same mathematical idea applied to some generalization of a utility function of high dimensionality; thus grandiose qualifiers. It's almost a creed. Yud's ex-wife is «Secretary of Global Optimization». Caroline Ellison's (FTX, SBF, EA) Tumblr is WorldOptimization. Scott Alexander's one-time proposal mission is to fix the world by slaying the «Moloch» of Darwinian processes and ushering in the world of maximum utility for the greatest number (presumably at the cost of immense temporary sin, if his serious fiction works much like Yud's. In any event it's good enough that I don't want to spoiler Unsong even more). AI doomer Zimmerman, too, recruits people into his mission of optimizing the world. I can't not blurt out that this is mostly another super-secular spin on Tikkun olam – multiplied by a smorgasboard of «neuroatypical» traits: subclinical-sociopathic minmaxing tendencies, autistic reification of abstract economic models, broken affective empathy, OCD-like neuroticism, love of abstract «elegance», systemizing SV tech startup energy, plus a bunch of other crap. Progressives are prone to interpret this as mere «tech bro» zeitgest, but tech bros are far more chill – and refreshingly egoistic; this Lewisean moral obsession with global optimization is a different beast.
At the final level, this Theme evolves from a handy analytic framework or a normative belief about running things to a hard prior about mechanisms by which competent, impressive things can run at all. Every thought is colored by utility functions; every decision is interpreted in light of optimizing for some value X; a powerful intelligence is assumed to be a consequentialist decisionmaker with a unitary utility function like a boardgame-playing adversarially trained AI is, and the prime fear is that in the process of maximizing said objective function it will also discover objective Instrumental Values – power, intelligence, and the magic of recursive self-improvement to get smarter to seize more power to…
It's not all narrative, they do have some proofs that apply to certain hypothetical AI setups, notably to Yudkowsky's original proposal, if only it were specified rigorously enough to be implemented and not just fail. This is a very rare case of Yud admitting a mistake. (Though only in retrospect; his later SIAI/Seed AI idea was, I think, also catastrophic yet vigorously pursued, and while he claims he conscientiously abstained from writing code, it looks more like his fancy language project has gone nowhere).
But it does not apply to human general intelligence, or to the most impressive swing at AGI we have come up with to date; and they began thinking in these terms long before finding any evidence for them. I posit it's because these people identify their consequentialism with being smart enough to «decouple» from soppy plebeian contexts and directly maximize the important value. I think it's simpler and cruder: they value intelligence due to having accrued what societal power they have through its demonstration, and from there it's just leaky associative reasoning. Yud, specifically, has no power or prestige or much of anything at all without his perceived intelligence, so being wrong and being dead are close in his mindspace.
The third Theme is just the synthesis of the first two: it's recursive self-improvement.
I believe it is Yud's philosophy proper, its specific thesis. It is really very compact, for all that he has written: empowerment, with Yud as the overseer.
The ethos of it is encapsulated in the slogan Tsuyoku Naritai!, and its theory, the source of much hope and fear, in the idea of a proper mind being a human-interpretable bag of functional parts.
Said parts may be many and tricky and interacting in confusing ways, like rulings of Talmud are, or the modular brain in Yud's preferred – and wrong – interpretation of neuroscience is; but it is non-negotiable that they be things understandable to Yud and, less importantly, the agent itself; not some illegible messy product of running simple general learning algorithms on a universal substrate.
This, Coherent Extrapolated Volition.
Thus, the Seed AI research program, the pursuit of self-rewriting AI in the apparent fashion of Lisp scripts.
Thus, Overcoming Bias and becoming LessWrong towards Methods of Rationality (which in practice are tossed aside when Harry or Yud are having an intuitive epiphany) and beyond, becoming stronger – not just to shrug off biased thoughts, but to rise above the unblemished baseline; and eventually, yes, build the first superhuman AI, and have it rebuild you into the form you will become worthy of.
All this recursion is circling the drain in very tight loops.
Thus, on the other hand, the contempt for data- and compute-intensive paradigm of artificial neural networks, for those filthy alien «inscrutable giant walls of floating-point numbers». For connectionists' easy attitude to non-interpretability, for the notion of emergence and for their sober observations that we have mastered too many skills non-existent in the ancestral environment to expert great catch when searching for function-specific modules. Classical era Yud dunks far, far more on connectionism than on GOFAI, and strawmans more; he reviled ANNs even when he believed them to be a dead end.
Sutton's Bitter Lesson, too, is anathema to him:
No, no, no, we are building a golem, Harry:
Do you have a concrete argument against recursive self-improvement? We've already got demonstrated capacities in AI writing code and AI improving chip design, isn't it reasonable that AI will soon be capable of rapid recursive self-improvement? It seems reasonable that AI could improve compute significantly or enhance training algorithms, or fabricate better data for its successors to be trained upon.
Recursive self-improvement is the primary thing that makes AI threatening and dangerous in and of itself (or those who control it). I too think Yudkowsky's desire to dominate and control AI development is dangerous, a monopolist danger. But he clearly hasn't succeeded in any grand plan to social-engineer his way into AI development and control it, his social skills are highly specialized and only work on certain kinds of people.
So are you saying that recursive self-improvement won't happen, or that Yud's model is designed to play up the dangers of self-improvement?
I reject that I need to prove something as logically impossible to ward off Yud's insistence that it's inevitable and justifies tyranny. This is sectarian bullshit and I'll address it in the text if I ever finish it. I think it's very relevant that his idea of proper scientific process is literally this:
...
This guy has done fuck all in his life other than read, and write, and think. He has never been graded by a mean professor, never been regularized by shame and inadequacy in a class of other bright kids, never stooped to empirical science or engineering or business or normal employment, never really grokked the difference between the map and the territory. He has an unrealistically, wildly inflated impression of how powerful an intelligence contorted into a Hofstadterian loop is. He has infected other geeks with it.
Recursive self-improvement doesn't work very well. Rationalists become cranks, AIs degenerate. As for better ideas, see around here. It is certain that we can improve somewhat, I think. In the limit, we will get an ASI from a closed experimental loop. That really is like creating a separate accelerated civilization.
But with ANNs, unlike Lisp scripts, it seems to require a great deal of compute, and compute doesn't just lie on the sidewalk. Yud thinks an AGI will just hack into whatever it wants, but that's a very sci-fi idea from 1990s; something he, I believe, dreamed to implement in the way already described – a singleton in the world of worthless meat sacks and classical programs. If you hack into an AWS cluster today to do your meta-learning training run, you'll suspend thousands of workloads including Midjourney pics and hentai (that people …check in real time), and send alarms off immediately. If you hack into it tomorrow, you'll get backtracked by an LLM-powered firewall.
No, I'm not too worried about an orthodox Yuddite self-improving AI.
You really can just siphon money out of the internet - people do it all the time to banks, in crypto, scams, social engineering and so on. Steal money, buy compute. Our AI could buy whatever it needs with stolen money, or it could work for its money, or its owners could buy more compute for it on the very reasonable assumption that this is the highest yielding investment in human history. We live in a service economy, bodies are not needed for a great deal of our work.
Say our AI costs 10 million dollars a day to run, (ChatGPT as a whole costs about 700K). 10 million dollars a day is peanuts in the global economy. Global cybercrime costs an enormous amount of money, 6 trillion a year. I imagine most of that cost includes the cost of fortifying websites, training people, fixing damage or whatever and only a small fraction is stolen. Even so, our AI needs only to grab 1% of that revenue and launder it to fund itself. This is not difficult. People do it all the time. And compute costs are falling, some smallish programs are being run on Macbooks as you explained earlier.
The danger is that somebody starts off with a weak superintelligence, perhaps from a closed experimental loop such as you nominate. Then it becomes a strong superintelligence rapidly by buying compute, developing architectural improvements and so on. Either it is controlled by some clique of programmers, bureaucrats or whatever (I think we both agree that this is a bad outcome) or it runs loose (also a bad outcome). The only good outcome is if progress is slow enough that power is distributed between the US, China, EU, hackers and enthusiasts and whoever else, that nobody gets a decisive strategic advantage. Recursive self-improvement in any meaningful form is catastrophic for humanity.
I think this means that you agree that superintelligences can recursively self-improve, that they're akin to another superintelligence? Then don't we agree?
Anyway, the authorities are extremely dopey, slow and stupid. The much vaunted US semiconductor sanctions against China meant that they simply... rented US compute to train their programs. Apparently stopping this is too hard for the all-powerful, all-knowing, invincible US government leviathan.
https://www.ft.com/content/9706c917-6440-4fa9-b588-b18fbc1503b9
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link