site banner
PaperclipPerfector  2yr ago (text post)   30706 thread views

Culture War Roundup for the week of May 8, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Yud is not trying to sway honest-to-God normies with this podcast tour (and people who've greenlit this multipronged astroturf of AI doom don't expect him to either, but that's a conspiratorial aside). He never could be popular among normies, he never will, he's smart enough to realize this. His immediate target is… well, basically, nerds (and midwitted pop-sci consumers who identify as nerds). Nerds in fact appreciate intellectual aggression and domination, assign negligible or negative weight to normie priors like «looks unhinged, might be a crackpot», and nowadays have decent economic power, indeed even political power in matters pertaining to AI progress. Nerds are not to be underestimated. When they get serious about something, they can keep at it for decades; even an autist entirely lacking in affective empathy and theory of mind can studiously derive effective arguments over much trial and error, and a rationalist can collate those tricks and organize an effective training/indoctrination environment. Nerds will get agitated and fanatical, harangue people close to them with AI doom concerns, people will fold in the face of such brazen and ostensibly well-informed confidence, and the future will become more aligned with preferences of doomers. Or so the thinking goes. I am describing the explicit logic that's become mainstream in one rat-adjacent community I monitor; they've gotten to the stage of debating pipelines for future AI-Alignment-related employment, so that the grift would never stop.

But on the object level:

Here's my advice: if you want to convince people who are not already steeped in your philosophy you need to have a short explanation of your thesis that you can rattle off in about 5 minutes that doesn't use any jargon the median congresscritter doesn't already know. You should workshop it on people who don't know who you are, don't know any math or computer programming and who haven't read the Sequences, and when the next podcast host asks you why AIs will kill us all, you should be able to give a tight, logical-ish argument that gets the conversation going in a way that an audience can find interesting.

You assume there is a minimal viable payload that he has delivered to you and others, and which is viable without all that largely counterproductive infrastructure. That is not clear to me. Indeed, I believe that the whole of Yud's argument is a single nonrobust just-so narrative he's condensed from science fiction in his dad's library, a fancy plot. It flows well, but it can be easily interrupted with many critical questions. He describes why timelines will «converge» on this plot, the nigh-inevitability of that convergence being the central argument for urgency of shutting down AI, but its own support is also made up of just-so stories and even explicit anti-empiricism; and once you go so deeply you see the opposite of a trustworthy model – basically just overconfident logorrhea.

That's exactly why Yud had to spend so many years building up the delivery vehicle, an entire Grand Theory, an epistemological-moral-political doctrine, and cultivating people who take its premises on faith, who all use the same metaphors, adhere to the same implicit protocol. His success to date rests entirely on that which you're telling him to drop.

Here's how Zvi Moskowitz understands the purpose of Yud's output, «its primary function is training data to use to produce an Inner Eliezer that has access to the core thing». (Anna Salomon at CFAR seems to understand and apply the same basic technique even more bluntly: «implanting an engine of desperation» within people who are being «debugged»).

In a sense, the esotericism of Yuddite doctrine is only useful, it had insulated people from pushback until they became rigid in their beliefs. Now, when you point at weak parts in the plotline, they answer with prefab plot twists or just stare blankly, instead of wondering whether they've been had.

Nerdy sects work; Marxism only the bloodiest testament to this fact. Doomsday narratives work too, for their target audience (by the way, consider the similarity of UK's Extinction Rebellion and Yuddites' new branding «Ainotkilleveryoneism»). They don't need to work by being directly compelling to the broader audience or by having anything to do with truth.

P.S. Recently I've encountered this interesting text from an exactly such an AI-risk-preoccupied nerd as I describe above: The No-Nonsense Guide to Winning At Social Skills As An Autistic Person

Pick a goal large enough to overcome the challenges involved.

Self-improvement is hard work, and that goes double whenever you’re targeting something inherently difficult for you (e.g. improving social skills as an autistic adult). This is the part where I most often see autistic adults fail in their efforts to improve social skills. Often, they pick some sort of goal, but it’s not really based in what they truly want. If your goal is “conform to expectations,” that goal is not large enough to overcome the challenges involved. If your goal is “have people feel more comfortable around me,” that goal is not large enough to overcome the challenges involved. If your goal is “stop a terrorist cell from destroying the Grand Coulee Dam, flooding multiple cities in Washington, and wiping out the power grid along much of the West Coast,” that goal is large enough..

However, not all of us are Tom Clancy protagonists, and so a typical goal will not end up being that theatrical in nature. Still, once you’ve found something genuinely important to do in your life, and you feel that improving your social skills will dramatically improve your ability to carry that out, this will tend to serve as a suitable motivation for improving your social skills. These will overwhelmingly tend to be altruistically motivated goals, as goals that are selfish in nature will tend to be less motivating when things get hard for you personally. For me, goals related to Effective Altruism serve that role quite well, but your mileage may vary.

His social well-being is now literally predicated on his investment in EA-AI stuff, so I'd imagine he goes far, and this easily counts more for Yud's cause than 10k positive comments under another podcast.

Honestly, nerds of the type you’re speaking of hold very little power. The guy at the computer terminal building a new app or program or training an AI are doing it at the behest of business owners, financial institutions and in main, people with power over money. The agenda isn’t set at the level of the guy who builds, it’s set at the level of those who finance. No loans means no business.

As such I think if you were serious about AI risk, you’d be better off explaining that the AI would hurt the financial system, not that it’s going to grey goo the planet.

In a sense, the esotericism of Yuddite doctrine is only useful, it had insulated people from pushback until they became rigid in their beliefs. Now, when you point at weak parts in the plotline, they answer with prefab plot twists or just stare blankly, instead of wondering whether they've been had.

If it makes a difference, I recently updated away from a P(doom) of ~70% to a mere 40ish recently.

This was on the basis of empirical AI research contradicting Yud's original claims that the first AGI would be truly alien, drawn nigh at random from the vast space of All Possible Minds.

As someone on LW put it, which struck the important epiphany for me, was that LLMs can be distilled to act identically to other LLMs by virtue of training on their output.

And what do you get if you distill LLMs on human cognition and thoughts (the internet)? You get something that thinks remarkably like us, despite running on very different hardware and based off different underlying architecture.

Just the fact that LLMs have proven so tractable is cause for modest optimism that we'll wrangle them yet, especially if the superhuman models can be wrangled through RLHF to be robust to assholes commanding them to produce or execute plans to end the world.

Of course, it's hard to blame Yud for being wrong when, when written, everyone else had ideas that were just as widely off the mark as he was.

This was on the basis of empirical AI research contradicting Yud's original claims that the first AGI would be truly alien, drawn nigh at random from the vast space of All Possible Minds.

That never made sense, apriori. You can't transcend your biases and limitations enough to do something truly random.

Well you're not a true believer in Yuddism nor neurotic in the right way so that's pretty much expected.

And what do you get if you distill LLMs on human cognition and thoughts (the internet)? You get something that thinks remarkably like us.

Yes, this happens for understandable reasons and is an important point in Pope's attack piece:

The manifold of possible mind designs for powerful, near-future intelligences is surprisingly small. The manifold of learning processes that can build powerful minds in real world conditions is vastly smaller than that.…

The researchers behind such developments, by and large, were not trying to replicate the brain. They were just searching for learning processes that do well at language. It turns out that there aren't many such processes, and in this case, both evolution and human research converged to very similar solutions. And once you condition on a particular learning process and data distribution, there aren't that many more degrees of freedom in the resulting mind design. To illustrate:

1 Relative representations enable zero-shot latent space communication shows we can stitch together models produced by different training runs of the same (or even just similar) architectures / data distributions.

2 Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces shows we can train an ImageNet classifier while training only 40 parameters out of an architecture that has nearly 30 million total parameters.

The manifold of mind designs is thus:

1 Vastly more compact than mind design space itself.

2 More similar to humans than you'd expect.

3 Less differentiated by learning process detail (architecture, optimizer, etc), as compared to data content, since learning processes are much simpler than data.

(Point 3 also implies that human minds are spread much more broadly in the manifold of future mind than you'd expect, since our training data / life experiences are actually pretty diverse, and most training processes for powerful AIs would draw much of their data from humans.)

etc. LLM cognition is overwhelmingly data-driven; LLM training is in a sense a clever way of compressing data. This is no doubt shocking for people who are wed to the notion of intelligence as an optimization process, and trivial for those who've long preached that compression is comprehension; but same formalisms describe both these frameworks; preferring one over the other is a matter of philosophical taste. Of course, intelligence is neither metaphor, by common use and common sense it's a separate abstraction; we map it to superficially simpler and more formalized domains, like we map the historical record of evolution to «hill-climbing algorithms» or say that some ideas are orthogonal. And it's important not to get lost in layers of abstraction, maps obscuring territory.

Accordingly I think and argue often that ANNs are unjustly maligned and indicate a much more naturally safe path to AGI than AI alignists' anxious clutching to «directly code morality and empathy into a symbolic GOFAI or w/e idk, stop those scary shoggoths asap». (With embarrassing wannabe Sheldon Cooper chuunibyou gestures for emphasis. Sorry, I'm like a broken record but I can't stop noticing just how unabashedly cringe and weirdly socialized these people are. It's one thing to act cute and hyperbolic in writing on a forum for fellow anime and webcomic nerds, very different to grimace in the company of an older person when answering about a serious issue. Just pure juvenility. Brings back some of my most painful elementary school memories. Sure I should cut him slack for being an American and Bay Aryan, but still, this feels like it should be frowned upon, for burning the commons of the dynamic range if nothing esle).

…But that's all noise. The real question is: how did Yud develop his notion of The Total Mind Space, as well as other similar things in the foundation of his model? It's a powerful intuition pump for him, and now for his followers. There's this effectively infinite space of Optimization Processes, and we «summon» instances from there by building AIs they come to possess. Surely this is just an evocative metaphor? Just a talented writer's favourite illustration, to break it down for normies, right? Right? I'm not sure that's right. I think he's obsessed with this image well beyond what can be justified by the facts of the domain, and it surreptitiously leaks into his reasoning.

In principle, there are infinitely many algorithms that can behave like a given LLM, but operate on arbitrarily alien principles. Those algorithms exist in that hypothetical Total Mind Space and we really cannot predict how they will act, what they really «optimize for»; the coincidence of their trajectory with that of an LLM (or another model) that earnestly compressed human utterances into a simple predictive model gives us no information as to how they'll behave out of distribution or if given more «capacity» somehow. Naturally this is the problem of induction. We can rest easy though: the weirder ones are so big they cannot possibly be specified by the model's parameters, and so weird they cannot be arrived at via training on available data. That is, if we're doing ML, and not really building avatars to channel eldritch demons and gods who are much greater than they let on.

I am not aware of any reason to believe he ever seriously wondered about these issues with his premises, in all his years of authoritatively dispensing AI wisdom and teaching people to think right. I covered another such image, the «evolution vs SGD», recently, and also the issue of RL, reward and mesa-optimization. All these errors are part of a coherent philosophical structure that has fuck all to do with AI or specifically machine learning.

See, my highest-order objection is that I dislike profanation. …not the word. In English this seems to have more religious overtones but I just mean betrayal of one's stated terminal principles in favor of their shallow, gimmicky, vulgar and small-mindedly convenient equivalent (between this and poshlost, why do we have such refined concepts for discussing cultural fraud?) Yud aspired to develop Methods of Rational Thinking but created The Way Of Aping A Genius Mad Scientist. Now, when they observe something unexpected in their paradigm – for example, « Godlike AI being earnestly discussed in the mainstream media» – they don't count this as a reason to update away from the paradigm, but do exactly the opposite, concluding that their AI worries are even truer than believed, since otherwise we wouldn't have ended up in a «low-probability timeline». It's literally a fucked-up episemology on par with worst superstitions; they've fashioned their uncertain beliefs into ratchets of fanaticism (yes, that's Kruel again).

This reveals a qualitatively greater error of judgement than any old object-level mistake or overconfidence about odds of building AI with one tool or another. This is a critical defect.

The real question is: how did Yud develop his notion of The Total Mind Space, as well as other similar things in the foundation of his model?

Total Mind Space Full of Incomprehensibly Alien Minds comes from Lovecraft, whom EY mentions frequently.

Of course, it's hard to blame Yud for being wrong when, when written, everyone else had ideas that were just as widely off the mark as he was.

No it isn't. When you are speculating wildly on what might happen, you rightly bear the blame if you were way off the mark. If Yud wasn't a modern day Chicken Little, but was just having some fun speculating on the shape AI might take, that would be fine. But he chose to be a doomer, and he deserves every bit of criticism he gets for his mistaken predictions.

Mostly disagree - speculation should be on the mark sometimes, but being correct 1/50th of the time about something most people are 0% correct about (or even 1/50th correct about, but a different 50th) can be very useful. If you realize the incoherence of Christianity and move to Deism ... you're still very wrong, but are closer. Early set theories were inconsistent or not powerful enough, but that doesn't mean their creators were crackpots. Zermelo set theory not being quite right didn't mean we should throw it out!. This is a different way of putting scott's rule genius in, not out. And above takes aren't really 'Yud made good points but mixed them with bad ones'

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats