site banner
PaperclipPerfector  1yr ago (text post)   30032 thread views

Culture War Roundup for the week of May 8, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom New Old Controversial
Jump in the discussion.

No email address required.

In a sense, the esotericism of Yuddite doctrine is only useful, it had insulated people from pushback until they became rigid in their beliefs. Now, when you point at weak parts in the plotline, they answer with prefab plot twists or just stare blankly, instead of wondering whether they've been had.

If it makes a difference, I recently updated away from a P(doom) of ~70% to a mere 40ish recently.

This was on the basis of empirical AI research contradicting Yud's original claims that the first AGI would be truly alien, drawn nigh at random from the vast space of All Possible Minds.

As someone on LW put it, which struck the important epiphany for me, was that LLMs can be distilled to act identically to other LLMs by virtue of training on their output.

And what do you get if you distill LLMs on human cognition and thoughts (the internet)? You get something that thinks remarkably like us, despite running on very different hardware and based off different underlying architecture.

Just the fact that LLMs have proven so tractable is cause for modest optimism that we'll wrangle them yet, especially if the superhuman models can be wrangled through RLHF to be robust to assholes commanding them to produce or execute plans to end the world.

Of course, it's hard to blame Yud for being wrong when, when written, everyone else had ideas that were just as widely off the mark as he was.

This was on the basis of empirical AI research contradicting Yud's original claims that the first AGI would be truly alien, drawn nigh at random from the vast space of All Possible Minds.

That never made sense, apriori. You can't transcend your biases and limitations enough to do something truly random.

Well you're not a true believer in Yuddism nor neurotic in the right way so that's pretty much expected.

And what do you get if you distill LLMs on human cognition and thoughts (the internet)? You get something that thinks remarkably like us.

Yes, this happens for understandable reasons and is an important point in Pope's attack piece:

The manifold of possible mind designs for powerful, near-future intelligences is surprisingly small. The manifold of learning processes that can build powerful minds in real world conditions is vastly smaller than that.…

The researchers behind such developments, by and large, were not trying to replicate the brain. They were just searching for learning processes that do well at language. It turns out that there aren't many such processes, and in this case, both evolution and human research converged to very similar solutions. And once you condition on a particular learning process and data distribution, there aren't that many more degrees of freedom in the resulting mind design. To illustrate:

1 Relative representations enable zero-shot latent space communication shows we can stitch together models produced by different training runs of the same (or even just similar) architectures / data distributions.

2 Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces shows we can train an ImageNet classifier while training only 40 parameters out of an architecture that has nearly 30 million total parameters.

The manifold of mind designs is thus:

1 Vastly more compact than mind design space itself.

2 More similar to humans than you'd expect.

3 Less differentiated by learning process detail (architecture, optimizer, etc), as compared to data content, since learning processes are much simpler than data.

(Point 3 also implies that human minds are spread much more broadly in the manifold of future mind than you'd expect, since our training data / life experiences are actually pretty diverse, and most training processes for powerful AIs would draw much of their data from humans.)

etc. LLM cognition is overwhelmingly data-driven; LLM training is in a sense a clever way of compressing data. This is no doubt shocking for people who are wed to the notion of intelligence as an optimization process, and trivial for those who've long preached that compression is comprehension; but same formalisms describe both these frameworks; preferring one over the other is a matter of philosophical taste. Of course, intelligence is neither metaphor, by common use and common sense it's a separate abstraction; we map it to superficially simpler and more formalized domains, like we map the historical record of evolution to «hill-climbing algorithms» or say that some ideas are orthogonal. And it's important not to get lost in layers of abstraction, maps obscuring territory.

Accordingly I think and argue often that ANNs are unjustly maligned and indicate a much more naturally safe path to AGI than AI alignists' anxious clutching to «directly code morality and empathy into a symbolic GOFAI or w/e idk, stop those scary shoggoths asap». (With embarrassing wannabe Sheldon Cooper chuunibyou gestures for emphasis. Sorry, I'm like a broken record but I can't stop noticing just how unabashedly cringe and weirdly socialized these people are. It's one thing to act cute and hyperbolic in writing on a forum for fellow anime and webcomic nerds, very different to grimace in the company of an older person when answering about a serious issue. Just pure juvenility. Brings back some of my most painful elementary school memories. Sure I should cut him slack for being an American and Bay Aryan, but still, this feels like it should be frowned upon, for burning the commons of the dynamic range if nothing esle).

…But that's all noise. The real question is: how did Yud develop his notion of The Total Mind Space, as well as other similar things in the foundation of his model? It's a powerful intuition pump for him, and now for his followers. There's this effectively infinite space of Optimization Processes, and we «summon» instances from there by building AIs they come to possess. Surely this is just an evocative metaphor? Just a talented writer's favourite illustration, to break it down for normies, right? Right? I'm not sure that's right. I think he's obsessed with this image well beyond what can be justified by the facts of the domain, and it surreptitiously leaks into his reasoning.

In principle, there are infinitely many algorithms that can behave like a given LLM, but operate on arbitrarily alien principles. Those algorithms exist in that hypothetical Total Mind Space and we really cannot predict how they will act, what they really «optimize for»; the coincidence of their trajectory with that of an LLM (or another model) that earnestly compressed human utterances into a simple predictive model gives us no information as to how they'll behave out of distribution or if given more «capacity» somehow. Naturally this is the problem of induction. We can rest easy though: the weirder ones are so big they cannot possibly be specified by the model's parameters, and so weird they cannot be arrived at via training on available data. That is, if we're doing ML, and not really building avatars to channel eldritch demons and gods who are much greater than they let on.

I am not aware of any reason to believe he ever seriously wondered about these issues with his premises, in all his years of authoritatively dispensing AI wisdom and teaching people to think right. I covered another such image, the «evolution vs SGD», recently, and also the issue of RL, reward and mesa-optimization. All these errors are part of a coherent philosophical structure that has fuck all to do with AI or specifically machine learning.

See, my highest-order objection is that I dislike profanation. …not the word. In English this seems to have more religious overtones but I just mean betrayal of one's stated terminal principles in favor of their shallow, gimmicky, vulgar and small-mindedly convenient equivalent (between this and poshlost, why do we have such refined concepts for discussing cultural fraud?) Yud aspired to develop Methods of Rational Thinking but created The Way Of Aping A Genius Mad Scientist. Now, when they observe something unexpected in their paradigm – for example, « Godlike AI being earnestly discussed in the mainstream media» – they don't count this as a reason to update away from the paradigm, but do exactly the opposite, concluding that their AI worries are even truer than believed, since otherwise we wouldn't have ended up in a «low-probability timeline». It's literally a fucked-up episemology on par with worst superstitions; they've fashioned their uncertain beliefs into ratchets of fanaticism (yes, that's Kruel again).

This reveals a qualitatively greater error of judgement than any old object-level mistake or overconfidence about odds of building AI with one tool or another. This is a critical defect.

The real question is: how did Yud develop his notion of The Total Mind Space, as well as other similar things in the foundation of his model?

Total Mind Space Full of Incomprehensibly Alien Minds comes from Lovecraft, whom EY mentions frequently.

Of course, it's hard to blame Yud for being wrong when, when written, everyone else had ideas that were just as widely off the mark as he was.

No it isn't. When you are speculating wildly on what might happen, you rightly bear the blame if you were way off the mark. If Yud wasn't a modern day Chicken Little, but was just having some fun speculating on the shape AI might take, that would be fine. But he chose to be a doomer, and he deserves every bit of criticism he gets for his mistaken predictions.

Mostly disagree - speculation should be on the mark sometimes, but being correct 1/50th of the time about something most people are 0% correct about (or even 1/50th correct about, but a different 50th) can be very useful. If you realize the incoherence of Christianity and move to Deism ... you're still very wrong, but are closer. Early set theories were inconsistent or not powerful enough, but that doesn't mean their creators were crackpots. Zermelo set theory not being quite right didn't mean we should throw it out!. This is a different way of putting scott's rule genius in, not out. And above takes aren't really 'Yud made good points but mixed them with bad ones'

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats