site banner

Culture War Roundup for the week of March 27, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

11
Jump in the discussion.

No email address required.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this? That could put a hamper on any kind of FOOM scenario. Rather than crack some kind of grand theory of intelligence, a super intelligence would need to hijack trillions of dollars worth of computing resources to gain a competitive advantage and a correspondingly huge amount of training data.

That could put a hamper on any kind of FOOM scenario.

I believe that's partially Altman's argument for accelerating in the moment, the «short timelines – slow takeoff» policy as he puts it. LLMs are not nearly the perfect way to build intelligence – they're perhaps as clunky as we are, only in a different manner. But that's a blessing. They are decidedly non-agentic, they have trouble with qualitative self-improvement, they can interpret their code no more than we do, and they are fat. (Though they sure can supervise training, and LLaMA tunes show you might not need a lot of data to transfer a meaningfully different character with enhanced capabilities, especially if these things proliferate – a tiny LoRA plus a prompt and external address for extra info to include in context will allow a some new «Sydney» to spread like wildfire).

As for Yud: like @georgioz says, it's more sophisticated. Now he admits that scaling (and other tricks) clearly suffices to achieve some nontrivial capabilities – hence his recent insistence on shutting it all down, of course; he expects GPT-5-class models to be dangerous, if not FOOMing yet. In that fragment he says that, at least circa 2006, he did not distinguish neural networks, expert systems and evolutionary algorithms, which probably explains why he acted (and still acts) as a maverick tackling hitherto-unforeseen problems: if you ditch the lion's share of GOFAI and connectionism, you aren't left with a ton of prior art. Less charitably, he was just ignorant.

Recapitulating human brain evolution is computationally intractable, far as we know, so his retroactive concession is rather stingy. People like Hinton apparently knew in advance, for all this time, that with a million times more compute neural networks will learn well enough. But all Yud had to say back then was that it's stupid to hope to build intelligence «without understanding how intelligence works» and all he has to say now is that it's a «stupid thing for a species to do». His notion of understanding intelligence is not much more sophisticated than his political propositions – I gather he thinks it's to be some sort of modular crap with formulas (probably Bayes rule as the centerpiece) written out explicitly for some rudimentary machine to interpret, the mathematically rigorous Utility function capturing personal moral code of the developers, and so on, basically babby's first golem.

Back when he hoped to actually build something, he thought the following:

Seed AI means that - rather than trying to build a mind immediately capable of human-equivalent or transhuman reasoning - the goal is to build a mind capable of enhancing itself, and then re-enhancing itself with that higher intelligence, until the goal point is reached.

General intelligence itself is huge.  The human brain, created by millions of years of evolution, is composed of a hundred billion neurons connected by a hundred trillion synapses, forming more than a hundred neurologically distinguishable areas. We should not expect the problem of AI to be easy. Subproblems of cognition include attention, memory, association, abstraction, symbols, causality, subjunctivity, expectation, goals, actions, introspection, caching, and learning, to cite a non-exhaustive list. These features are not "emergent". They are complex functional adaptations, evolved systems with multiple components and sophisticated internal architectures,  whose functionality must be deliberately duplicated within an artificial mind. If done right, cognition can support the thoughts implementing abilities such as analysis, design, understanding, invention, self-awareness, and the other facets which together sum to an intelligent mind. …

seed AI also mistrusts that connectionist position which holds higher-level cognitive processes to be sacrosanct and opaque, off-limits to the human programmer, who is only allowed to fool around with neuron behaviors and training algorithms, and not the actual network patterns. Seed AI does prefer learned concepts to preprogrammed ones, since learned concepts are richer. Nonetheless, I think it's permissible, if risky, to preprogram concepts in order to bootstrap the AI to the point where it can learn. More to the point, it's okay to have an architecture where, even though the higher levels are stochastic or self-organizing or emergent or learned or whatever, the programmer can still see and modify what's going on. And it is necessary that the designer know what's happening on the higher levels, at least in general terms, because cognitive abilities are not emergent and do not happen by accident. Both classical AI and connectionist AI propose a kind of magic that avoids the difficulty of actually implementing the higher layers of cognition. Classical AI states that a LISP token named "goal" is a goal. Connectionist AI declares that it can all be done with neurons and training algorithms. Seed AI admits the necessity of confronting the problem directly.

(From another document)

The Singularity Institute seriously intends to build a true general intelligence, possessed of all the key subsystems of human intelligence, plus design features unique to Al. We do not hold that all the complex features of the human mind are "emergent", or that intelligence is the result of some simple architectural principle, or that general intelligence will appear if we simply add enough data or computing power. We are willing to do the work required to duplicate the massive complexity of human intelligence; to explore the functionality and behavior of each system and subsystem until we have a complete blueprint for a mind. … Our specific cognitive architecture and development plan forms our basis for answering questions such as "Will transhumans be friendly to humanity?" and "When will the Singularity occur?" At the Singularity Institute, we believe that the answer to the first question is "Yes" with respect to our proposed Al design - if we didn't believe that, the Singularity Institute would not exist. Our best guess for the timescale is that our final-stage Al will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

And:

If I were to try quantifying the level of brainpower necessary, my guess is that it's around the 10,000:1 or 100,000:1 level. This doesn't mean that everyone with a 160 IQ or 1600 SAT will fit the job, nor that anyone without a 160 IQ or 1600 SAT is disqualified. Standardized tests don't necessarily do a very good job of directly measuring the kind of horsepower we're looking for. On the other hand, it isn't very likely that the person we're looking for will have a 120 IQ or a 1400 on the SAT.

To be blunt: If you're not brilliant, you are basically out of luck on being an AI programmer. You can't put in extra work to make up for being nonbrilliant; on this project the brilliant will be putting in extra work to make up for being human. You can't take more time to do what others do easily, and you can't have someone supervise you until you get it right, because if the simple things have to be hammered in, you will probably never learn the complex things at all. …

You should have read through [Levels of Organization in General Intelligence] and understood it fully. The AI theory we will actually be using is deeper and less humanlike than the theory found in LOGI, but LOGI will still help you prepare for encountering it.

The four major food groups for an AI programmer:

  • Cognitive science
  • Evolutionary psychology
  • Information theory
  • Computer programming

Then a massive list of subdomains that is is basically a grab-bag of insight porn we've been awash in for the last two decades, presumably cultivating sparks of AGI in lesswrong and /r/slatestarcodex regulars.

The three-layer model of intelligence is necessary, but not sufficient. Building an AI "with sensory modalities, concepts, and thoughts" is no guarantee of intelligence. The AI must have the right sensory modalities, the _right_concepts, and the right thoughts.

Unfortunately it seems like linear algebra is just about enough.

Wait, Yudkowsky doesn't believe in the scaling hypothesis? That's super interesting to me! Has he written about this?

In this specific interview Yud said something along the lines that of course scaling is capable of producing general intelligence - in the end evolution did that blindly with human brain so it should be possible. He was just more sceptical regarding gains by more compute. Needless to say, he is less sceptical now.