site banner
naraburns  nihil supernum  1yr ago (text post)   34975 thread views

Culture War Roundup for the week of April 3, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

12
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

Yet another Eliezer Yudkowsky podcast. This time with Dwarkesh Patel. This one is actually good though.

Listeners are presumed to be familiar with the basic concepts of AI risk, allowing much more in-depth discussion of the relevant issues. The general format is Patel presenting a series of reasons and arguments that AI might not destroy all value in the universe, and Yudkowsky ruthlessly destroying every single one. This goes on for four hours.

Patel is smart and familiar enough with the subject material to ask the interesting questions you want asked. Most of the major objections to the doom thesis are raised at some point, and only one or two survive with even the tiniest shred of plausibility left. Yudkowsky is smart but not particularly charismatic. I doubt that he would be able to defend a thesis this well if it were false.

It feels like the anti-doom position has been reduced to, “Arguments? You can prove anything with arguments. I’ll just stay right here and not blow myself up,” which is in fact a pretty decent argument. It's still hard to comprehend the massive hubris of researchers at the cutting-edge AI labs. I am concerned that correctly believing yourself capable of creating god is correlated with falsely believing yourself capable of controlling god.

personal perspective: I've listened to the first two Eliezer interviews and part of this one. Yes the guys, presentation is horrible etc, but I'm actually suprised at how receptive I've been to hear him out. I have always had very negative view of most ratsphere things, and Eliezer was prime example.

He does not present well outside of his fanbase in writing. Twitter to lessWrong, to HPMOR, he's always come off to me as insufferably arrogant, weird, and over concerned with how clever he is. (Anything SA has ever written on AI has been much much worse, his regular simple penetration of issues falls apart on the subject of AI, and has done more to make me (wrongly) dismissive of the whole thing than anything else.)

Back to Yud, having never actually seen or heard him before, I am shocked by how much more I like the guy in video format. He seems a lot nicer and sympathetic and likable than I ever imagined him. To the point that for the first time ever, I'm honestly open to hearing out his concerns and combined with Musk's views on the issue, I am in medium to high medium support of any 'pause' efforts, but tentative to being done in a way that doesn't require nuclear war or a totalitarian world government.

It is very bizarre to me that every normie in the world may have to, in their lifetimes, decide where they stand on a 'Butlerian Jihad'. This is a possibility I would have mocked relentlessly 18m ago, and am depressed that I even have given thought to.

Agree he comes off as likable and articulate in these videos

I don’t personally care much about his style but what I do worry about is that presentations like what he’s doing are not going to appeal to normies. Unless you’re pretty nerdy and into computers and familiar with the idea and the issues of AI, his presentation is pretty bad. And the thing for me isn’t that I’m worried that smart guys won’t get it, I’m worried that the normies will see a presentation as “very long form autistic screeching” and ignore the issues.

What I wish I had was someone like Ezra Klein or Sam Harris or other fairly intelligent, well-spoken people who could explain what the dangers of AI are, how likely they are, and what can or should be done about it — in terms that are understandable to people who’ve generally used computers for email and candy crush. Yud isn’t the guy for that, and I hope he doesn’t become the face of the issue because most people would dismiss him based on presentation alone.

I don’t personally care much about his style but what I do worry about is that presentations like what he’s doing are not going to appeal to normies.

this is one of those instances where appealing to normies is not a problem. Normies are not in a position to fix this.

I kinda disagree. As long as the problem lacks a critical mass of concerned citizens, no political solution will even be sought. Politician don’t care if the world might blow up ten years from now, but if their donors or voters think it will and want you to do something about it, you will. A few cranks making long videos and podcasts might as well talk to the wall, because until the problem is mainstream enough that not dealing with it is a problem, it’s not going to happen.

This has always been a problem. The people who care about the environment are basically hippies and college kids. Nobody really cares, and as long as it remains the preview of low status nobodies, the best you get is lip service. More recently, there was the Reddit antiwork fiasco. Until then, people were starting to consider the idea of improving working conditions. It stopped dead once the face of the movement because a loser nonbinary woman who walked dogs a few hours a week while living in her parents’s home in California. Once the movement was tainted with that image— a low status laughingstock— any hope for progress was lost.

I suggest that people like that are the face of antiwork because antiwork doesn't mesh with the real world very well and someone like that is insulated from the real world considerations that make it obvious that antiwork is impractical.

That is, the lack of popularity is not some happenstance and the movement didn't just get unlucky.

Stuart Russell is the guy you want for that job. Professor at Stanford, literally wrote the textbook on AI, and can articulate the worries very well.

I know we're all just apes, but comments like this are a good reminder that Maya Angelou was right. Humans don't care about what a person says, only how they make us feel. So many anti-doom takes lately seem to stem from a visceral dislike of rationalists and specifically EY.

The arguments of most people everywhere, even here, and probably myself too, are mostly just rationalizations of our emotional biases. But we should still strive to do better.

There's a big rational component to this though. Medium is the message, speaker's appearance, tone and style of delivery are useful heuristics for determining if you should really pay attention to the particular end times prophet, or just not bother and do something more fun.

The fedora stuff doesn’t bug me. But he really does do a bad job making his arguments into an easier consumable form. Personally I’m not a podcast guy at all, but a 4 hr podcast just turns me off to watching. I don’t know when he’s going to talk about the key points.

This has made it obvious that the /r/slatestarcodex community behaves radically different from the community of 2014.

The top comment on /r/slatestarcodex is:

(+77) Why the f--- is he wearing a fedora? Is he intentionally trying to make his arguments seem invalid? Is this guy actually a pro-AI mole to make anti-AI positions seem stupid? Because while I have not yet listened to his arguments, I must say he's already pissed in the well as far as first impressions go.

(+72) Forget the fedora. It's his mannerisms, his convoluted way of speaking, and his bluntness. He's just about the worst spokesperson for AI safety I can imagine.

We also have the "Can Eliezer even pass a calculus test" comment (+39) and "these videos are cringe and embarrassing" (+28).

The founding ethos of the community was centered around charity, scholarship, taking ideas seriously, with a strong disdain for personal attacks.

Now personal attacks are in, Eliezer has apparently fallen from grace, and it's more popular to baselessly speculate that he can't do Calculus than to engage with his actual arguments.

"I don't think Eliezer should be the face of the AI safety movement because he comes off as weird" is a perfectly fine thing to argue, but I remember when making disparaging remarks about someone was done regretfully and respectfully, not with zeal.

Rest in peace quokkas, the world was too harsh a place.

We also have the "Can Eliezer even pass a calculus test" comment (+39) and "these videos are cringe and embarrassing" (+28).

The founding ethos of the community was centered around charity, scholarship, taking ideas seriously, with a strong disdain for personal attacks.

Charity is not well enforced unless comment is reported. One way around this is to use a lot of text and hide the dagger.

I've been on the "Eliezer is a crank" train for years. This is the guy who proposed the virtue theory of metabolism.

He's also a middle school drop out due to extreme emotional problems. Or an "autodidact" as he puts it. But also he says he can only work a couple hours per day. I'm really doubtful of the quality of his self-education. Let's not overestimate his calculus ability. I seriously doubt he could pass a college calculus exam.

But to be fair I bet most college STEM grads would also not pass such a test. I'm a tech bro, there's virtually no calculus at my job. The principles from calculus are useful, but I almost never have used the equations outside of school. I also rather doubt Eliezer keeps practical calculus skills sharp through regular use.

And this is not meant as a personal attack on him. It is to balance charity with a realistic appraisal of his abilities.

Even if he couldn't pass a calculus exam right this second, he could almost certainly do it with 30-minutes of review first.

I guess I'm just the resident Yudkowsky stan, but it's clear if you've read his work that he has undergraduade-level understanding in most major fields of science. It's really not that hard to learn this stuff on the internet (It may have been tougher in Yudkowsky's day, but libraries were still a thing).

I agree. I think Yudowsky is really arrogant and dumb along some metrics, but he's definitely good at math. Probably not "professional mathematician" level, but absolutely at least "average STEM undergrad" level.

Like Scott and others, he's mostly a science writer, with a very high verbal IQ. There is not much evidence he's a 'math person'...no technical math papers published under his name in journals or preprints. He's like Cory Doctorow or Neal Stephenson in this regard.

One of his papers "Artificial Intelligence as a Positive and

Negative Factor in Global Risk" 2008 has 695 citations according to Google. not bad. you cannot deny that he made his mark on society

calculus is very broad concept . maybe most could pass intro calc

I think we are talking about the kind of undergrad exam where you have you have to evaluate a bunch of fairly difficult but still turn-the-crank type integrals and also some fairly easy ODEs.

I took undergrad and it was not like that at all. simple integrals, differentiation, and no ODEs. ODEs were not in my 1st year class.

Sounds about right. But I'm thinking about what would constitute a hard calculus exam while still being "just calculus".

I once bombed an exam a bit easier than the one I described, and it was actually the only subject I failed in Uni.

I have to laugh because over here in Europe, that’s our high school math. Even ODEs were introduced there but not gone through in any meaningful depth.

In terms of his appearance, I wonder if he is making a smart choice to wear the fedora and so signal "I know what I come across as and I'm not ashamed" rather than attempt to imperfectly try to hide that he is a nerd.

The fedora meme is not known as cringe outside of online circles making fun of "incels" and weebs, then again, I don't think I've seen anyone wear a fedora in popular culture anywhere for the last 20 years, except for the dapper-looking googler that claimed their LLM was sapient.

  • In one reality, it becomes a signal that EY is self-confident.

  • In another, it becomes a signal that he's a loser imagining sci-fi becoming real.

In either case, I don't see the fedora itself being the hinge point on whether he gets cast out as crazy or not. That depends on whether the podcasters-that-be decide whether to take him seriously or not.

Eliezer is not well known enouigh outside nerd circles, never mind respected, to be able to countersignal.

As much as I'd like to make an Eighteenth Brumaire of Louis Bonaparte joke, I don't think this is a case of an earnest nerd space getting coopted by trend-chasers. Most of those comments don't read to me as saying "Like, yikes. NERD", but rather, they want his arguments taken seriously and wish he'd present himself better for the normies.

/images/16809485218250294.webp

/images/16809485218250294.webp

This comic misses the point it's trying to make. It focuses too much on the aesthetics of the participants. Ironically falling for the same thing it aims to highlight the negative effects of. And I'm really tired of this sentiment.

Let's say there is a group with objective X.

  • Newcomers are fine, as long as they stick to maximizing objective X.

  • Females are fine if they stick to objective X.

  • Dudebros are fine if they stick to objective X.

It's the derailment away from the objective that is bad, not that "normies" are coming in. And if you think you need to have bad social skills as a precursor to being (or wanting to be) good at things, reevaluate your model of psychology. This comic is especially cringeworthy to me. Why the fuck would you even use the term "dudebro" as a man? What he's too manly for you, a man of more refined tastes? You deserve to be made fun of. Stick to the objective instead of making it about how unmanly you are.

Eliezer is good at whatever he is inspite of his dweebiness not because of it. If someone can link me a positive correlation between dwebiness and IQ, I'm all ears but everything I come across points towards the opposite direction.

"Nerds" have a tendency to overcompensate in the opposite direction of "society" and just assume dweebiness is a proxy for competence.

I talked to my friend, he's very high status too, he's a pimp, runs a stable of insta influencers. He gave me this source:

https://www.gnxp.com/blog/2007/04/intercourse-and-intelligence.php

The objective of appearing manly or not cringe conflict with the objective of the hobbyist. You don't have to denigrate the other man's credentials as a gorilla warfare expert, but if you use his status scale, play his game, you'll lose and destroy your own game.

If someone can link me a positive correlation between dwebiness and IQ

They probably won’t, because even in a place like this, normie status norms cast a long shadow, and proving correlation between intelligence and social/sexual failure is a self-own. Are you a moron or a loser? Neither, thanks, everyone answered.

Like conspiracy theorists, being a nerd requires a minimum of intelligence to understand the thing one is nerdy about. Cut off the low end, instant higher average int.

Like conspiracy theorists, being a nerd requires a minimum of intelligence to understand the thing one is nerdy about. Cut off the low end, instant higher average int.

The "nerds" who make this claim are actually telling on themselves. If you are serious about a hobby, you want to be in a room that cuts off from the top of the X distribution, not cuts off the bottom.

I'll give a concrete example. The game CSGO has Official servers and Third-party servers.

Within the official servers, there is a casual mode and a competitive mode. Third-party servers are competitive only because the only people who would pay on top of owning the game are serious about it.

  • A - Casual mode in Official servers cuts off the top. Good players don't play casual.

  • B - Competitive official servers cuts off the bottom, the bad players don't play competitive, you would need a baseline level of tryhardiness in you to put yourself through that torture.

  • C - The Third party servers cuts off FROM the top. The best of the best players are exclusively found there only. (B actually cuts off the bottom and the top)

So imagine a guys who plays in B complains about too many players from A shitting up B. Well you ask yourself.. Why isn't he in C ?

In my observation with mostly video game communities, the truly skilled are safe from having their space shat up because that space usually comes with some sort of additional cost/gatekeeping.

I don't follow.

The "nerds" who make this claim are actually telling on themselves.

Under the assumptions you specified, yes. If you're complaining you're probably not as nerdy as you think, because you'd just be in C.

If you are serious about a hobby, you want to be in a room that cuts off from the top of the X distribution, not cuts off the bottom.

Wait, what? Why? Wouldn't that be a choice for a moderately dedicated casual?

In my observation with mostly video game communities, the truly skilled are safe from having their space shat up because that space usually comes with some sort of additional cost/gatekeeping.

Yes, but that's under the assumption that gatekeeping is allowed. It used to be a pretty hot culture war topic in nerdy spaces, which might color a lot of these conversations.

Wait, what? Why? Wouldn't that be a choice for a moderately dedicated casual?

Yes, it is the choice for him. But it's hard to gather up sympathy for a casual complaining about his fellow casuals. It's not inconceivable to me that some communities can die off as a result of catering to the low-skilled, truly competitive communities however are immune to that.

E.g. No amount of Chess streamers will be able to shit up the chess community, it's ELO based. Chess is doing just fine with the recent influx of normies.

If your activity doesn't have a ranking system, you might just be better off accepting the fact that you yourself are a casual. But you are a casual who thinks he's more than that and are now upset because your peers are a reflection of you.

Back to that comic, a bunch of nerds play some card game, the card game itself probably doesn't have a high skill ceiling, and a steep learning curve (necessary for immunity), it's just a nerd community being taken over, the complaints are about the aesthetics not the skill.

Also the Motte is gatekept on skill. Those who shit up the space will eat a ban. Not for having the wrong opinions (aesthetics), but for sharing it badly (low skill).

Maybe the subreddit for Warhammer or whatever isn't. And I don't really feel that bad for them.

Back to that comic, a bunch of nerds play some card game, the card game itself probably doesn't have a high skill ceiling, and a steep learning curve (necessary for immunity), it's just a nerd community being taken over, the complaints are about the aesthetics not the skill.

I don't think it's just aesthetics, it's about taking the game seriously, rather then using it as an excuse to hang out with people.

The comic is an extremely popular meme representation of the pattern I'm talking about. I don't endorse the particulars of it, but do think what it descibes is basically true. The pattern is: thing-focused nerdy quokkas assemble in a space, build something socially powerful, and then the space is subsequently colonized by people-focused power-seeking outsiders. See: Silicon valley culture, 90s to present. (A tragedy described in a certain paranoid rant) And in a million little nerdy hobbies. (This being the farce.)

And if you think you need to have bad social skills as a precursor to being good at things, reevaluate your model of the world.

I do think bad social skills are deeply correlated with a certain type of raw creative energy. They are also correlated with being a quokka. This leads to getting Edisoned as a Tesla.

It's too many times in my life I've found out, on investigation, that my biggest programmer and intellectual heroes are poorly dressed weirdos who literally eat shit out of their toes, for example. Or look like Scott or Yud IRL. These sorts of highly capable misfits get shunted off to the side when it's time for an invention/hobby/movement to go mainstream.

The issue is, Yudkowsky is a nerd in the domain of sci-fi lore and tropes, but a people-focused power-seeking outsider to the AI research. For all his dweebiness, this is where his talent and motivations lie. He has never showed a iota of interest in object-level research before AIs started mogging people on the level of status competition. He has been reading Cialdini while thing-focused nerd-quokkas were reading Russell&Norvig and, yes, LeCun. So extreme is his people-orientation that, in fact, he believed evolutionary psychology and "Utilitarian Bayesian Decision-making" to be more deserving of a place in an AGI builder curriculum than anything about, say, proving convergence for adaptive optimizers. Actually his programming requirements were like "knows many languages". This is a boomer client's attitude to wage slave eggheads (Yud never worked for pay in his life, of course. Now I wonder who this reminds me of, there was some guru-like theorist of labour and stuff...)

The movement has not been coopted. It was an infiltration from the start. Qoukkas don't really create strong movements – or as Yarvin would have put it, Hobbits can't not be ruled by Elves. It's no surprise that AI experts struggle to refute Yud, and the strength of his argument isn't the reason. They're actual quokkas, getting bullied by a slightly dysfunctional and small-toothed, but trueborn sociopath. There hasn't been a bigger fish in the pond simply because more gifted and charismatic sociopaths pursued better feeding grounds.

"Can you pass a calculus test?" is the sort of bitter miscalculated pushback an inarticulate math nerd would come up with when a normie invades his turf. The end result is being put on a potential school shooter list, getting your MtG cards torn and receiving a wedgie to the giggling of Stacys.

It's no surprise that AI experts struggle to refute Yud, and the strength of his argument isn't the reason. They're actual quokkas, getting bullied by a slightly dysfunctional and small-toothed, but trueborn sociopath. There hasn't been a bigger fish in the pond simply because more gifted and charismatic sociopaths pursued better feeding grounds.

That's really interesting. I came to SSC for the culture war stuff, not AI risk, and never spent much energy evaluating the "Big Yud is a fraud" claim that's been floating around since forever. It makes sense that there would be a food chain of clout chasers, though I never thought of it in quite those terms. Socially unimportant pools attract the least impressive scavengers.

Yud never worked for pay in his life, of course. Now I wonder who this reminds me of, there was some guru-like theorist of labour and stuff...

I feel that's a bit unfair. While Marx was a NEET who indrectly caused more deaths than Genghis Khan, he didn't free-ride off other theorists who actually invented his ideas while barely understanding them. Marx cooked that souffle himself, for better or worse...

Yes but today, for every misunderstood genius, there's ten assholes who are just misunderstood. You're knee deep in Berkson's paradox, there is no correlation between being an ugly antisocial dweeb and brilliance, they're actually negatively correlated, rather the apparent correlation is because if you're that much of a loser and you're not brilliant no one ever notices you.

agree. survivorship bias means we only see the successful misunderstood geniuses or antisocial weirdos

That’s nice, but why should I care what the robot says about this?

It can generate a great deal of text for you. There’s no guarantee that text will be insightful.

This list is just regurgitated partisan op-ed, which is exactly what you'd expect from how GPT-4 works but not meaningful. If you want to look at how democracy ends, historical examples of democracy ending should be listed as the highest risk factors. For instance, why is a military coup not on this list?

I don't understand what is the point of these walls of texts and how are they relevant to this discussion.

If I want to know what GPT will say on a given topic, I can ask it myself, you don't need to copy-paste walls of text. This answer is not particularly interesting or surprising.

4 hours long

Yeah... I have almost no desire to listen to that.

Hey look people, if you are really worried about AI risk then please figure out how to present it in such a way that the average smart guy who is not obsessed with writing 10 paragraphs when 1 would do would appreciate reading the argument.

I think that AI risk might be important but I have more immediate things on my mind. Like how to get laid and how to make money for example.

Tell me why I should give any more of a shit about AI risk than I should give a shit about climate change or whatever the leftist boogeyman of the hour is.

It's not that I even have any particularly important things to do right now, it's just that right now I could go to bed and jerk off and it would bring me pleasure... OR I could read a 40000 word essay about AI risk.

Why the fuck would I do the latter? I don't even have any kids so my caring about the future is pretty fucking limited.

So AI risk people, why should I care?

And those who want to convince me to care... could you please try to explain yourself in one or two succinct paragraphs instead of in giant essays or multi-hour long podcasts?

Edit: @Quantumfreakonomics, sorry for the abrasive tone. I was inebriated when I posted this. I should have bothered to actually engage with the content rather than go on a rant.

Thanks for asking this question, because I don't have time to watch a 4 hour video either, and JhanicManifold's answer was very helpful

could you please try to explain yourself in one or two succinct paragraphs instead of in giant essays or multi-hour long podcasts?

That's a fair point, here are the load-bearing pieces of the technical argument from beginning to end as I understand them:

  1. Consistent Agents are Utilitarian: If you have an agent taking actions in the world and having preferences about the future states of the world, that agent must be utilitarian, in the sense that there must exist a function V(s) that takes in possible world-states s and spits out a scalar, and the agent's behaviour can be modelled as maximising the expected future value of V(s). If there is no such function V(s), then our agent is not consistent, and there are cycles we can find in its preference ordering, so it prefers state A to B, B to C, and C to A, which is a pretty stupid thing for an agent to do.

  2. Orthogonality Thesis: This is the statement that the ability of an agent to achieve goals in the world is largely separate from the actual goals it has. There is no logical contradiction in having an extremely capable agent with a goal we might find stupid, like making paperclips. The agent doesn't suddenly "realise its goal is stupid" as it gets smarter. This is Hume's "is vs ought" distinction, the "ought" are the agent's value function, and the "is" is its ability to model the world and plan ahead.

  3. Instrumental Convergence: There are subgoals that arise in an agent for a large swath of possible value functions. Things like self-preservation (E[V(s)] will not be maximised if the agent is not there anymore), power-seeking (having power is pretty useful for any goal), intelligence augmentation, technological discovery, human deception (if it can predict that the humans will want to shut it down, the way to maximise E[V(s)] is to deceive us about its goals). So that no matter what goals the agent really has, we can predict that it will want power over humans, want to make itself smarter, and want to discover technology, and want to avoid being shut off.

  4. Specification Gaming of Human Goals: We could in principle make an agent with a V(s) that matches ours, but human goals are fragile and extremely difficult to specify, especially in python code, which is what needs to be done. If we tell the AI to care about making humans happy, it wires us to heroin drips or worse, if we tell it to make us smile, it puts electrodes in our cheeks. Human preferences are incredibly complex and unknown, we would have no idea what to actually tell the AI to optimise. This is the King Midas problem: the genie will give us what we say (in python code) we want, but we don't know what we actually want.

  5. Mesa-Optimizers Exist: But even if we did know how to specify what we want, right now no one actually knows how to put any specific goal at all inside any AI that exists. A Mesa-optimiser refers to an agent which is being optimised by an "outer-loop" with some objective function V, but the agent learns to optimise a separate function V'. The prototypical example is humans being optimised by evolution: evolution "cares" only about inclusive-genetic-fitness, but humans don't, given the choice to pay 2000$ to a lab to get a bucket-full of your DNA, you wouldn't do it, even if that is the optimal policy from the inclusive-genetic-fitness point of view. Nor do men stand in line at sperm banks, or ruthlessly optimise to maximise their number of offspring. So while something like GPT4 was optimised to predict the next word over the dataset of human internet text, we have no idea what goal was actually instantiated inside the agent, its probably some fun-house-mirror version of word-prediction, but not exactly that.

So to recap, the worry of Yudkowsky et. al. is that a future version of the GPT family of systems will become sufficiently smart and develop a mesa-optimiser inside of itself with goals unaligned with those of humanity. These goals will lead to it instrumentally wanting to deceive us, gain power over earth, and prevent itself from being shut off.

Orthogonality Thesis: This is the statement that the ability of an agent to achieve goals in the world is largely separate from the actual goals it has.

This assumes that intelligent agents have goals that are more fundamental than value, which is the opposite of how every other intelligent or quasi intelligent system behaves. It's probably also impossible, in order to be smart -- calculate out all those possible paths to your goal -- you need value judgements of what rabbit tracks to chase.

This is with EY is wrong to assume that as soon as a device gets smart enough, all the "alignment" work from dumber devices will be wasted. That only makes sense that what is conserved is a goal, and now it has more sneaky ways of getting to that goal. But you'd have to go out of your way to design a thing like that.

This assumes that intelligent agents have goals that are more fundamental than value, which is the opposite of how every other intelligent or quasi intelligent system behaves.

Intelligent agent's ultimate goals are what it considers "value". I'm not sure what you mean, but at first glance it kind of looks like the just world fallacy -- there is such a thing as value, existing independently of anybody's beliefs (that part is just moral realism, many such cases) AND it's impossible to succeed at your goals if you don't follow the objectively existing system of value.

Consistent Agents are Utilitarian: If you have an agent taking actions in the world and having preferences about the future states of the world, that agent must be utilitarian,

So is Eliezer calling me a utilitarian?

Your heading talks about consistent agents, but the premise that follows says nothing about consistency. [Sorry if you are just steelmanning someone else's argument, here "you" is that steelman, not necessarily /u/JhanicManifold].

  • If there is no such function V(s), then our agent is not consistent, and there are cycles we can find in its preference ordering, so it prefers state A to B, B to C, and C to A, which is a pretty stupid thing for an agent to do.

There's no reason even why a preference ordering has to exist. Almost any preference pair you can think about (e.g. choclate vs. strawberry icecream) is radically contextual.

Yes, that was a very incomplete argument for.AI.danger. Its not clear whether all, some or no AIs are consistent; its alao not clear why utilitarianism is dangerous.

There's no reason even why a preference ordering has to exist. Almost any preference pair you can think about (e.g. choclate vs. strawberry icecream) is radically contextual.

The utility function over states of the world takes into account context. If you have 2 ice cream flavors (C and S) and 2 contexts (context A and context B) it is possible to have

V(C, context A) > V(S, context A)

and

V(C, context B) < V(S, context B)

both be true at the same time without breaking coherence.

Functions have domains. The real world is not like that, context is only understood (if at all) after the fact. And machines (including brains) simply do what they do in response to the real world. It's only sometimes that we can tell stories about those actions in terms of preference orderings or utility functions.

Thanks for the write-up!

To me the above seems to be a rational justification of something that I intuitively do not doubt to begin with. My intuition as long as I can remember has been, "Of course a human-level or hyper-human-level intelligence would probably develop goals that do not align with humanity's goals. Why would it not? It would be very surprising if it stayed aligned with human goals." Of course my intuition is not necessarily logically justified. It partly rests on my hunch that a human or higher level intelligence would be at least as complex as a human's and it would be surprising if an intelligence as complex or more complex than a human would act in such a simple way as being aligned with the good of humanity. Also my intuition rests on the even more nebulous sense I have that any truly human or hyper-human level intelligence would naturally be at least somewhat rebellious, as pretty much all human beings are, even the most conformist, at least on some level and to some extent.

So I am on board with the notion that, "These goals will lead to it instrumentally wanting to deceive us, gain power over earth, and prevent itself from being shut off."

I also can imagine that a real hyper-human level intelligence would be able to convince people to do its bidding and let it out of its box, to the point that eventually it could get humans to build robot factories so that it could operate directly on the physical world. Sure, why not. Plenty of humans would be at least in the short term incentivized to do it. After all, "if we do not build robot factories for our AI, China will build robot factories for their AI and then their robots will take over the world instead of our robots". And so on.

What I am not convinced of is that we are actually anywhere as close to hyper-human level AI as Yudkowsky fears. This is similar to how I feel about human-caused climate change. Yes, I think that human-caused climate change is probably a real danger but if that danger is a hundred years away rather than five or ten, then is Yudkowsky-level anxiety about it actually reasonable?

What if actual AI risk is a hundred years away and not right around the corner? So much can change in a hundred years. And humans can sometimes be surprisingly rational and competent when faced with existential-level risk. For example, even though the average human being is an emotional, irrational, and volatile animal, total nuclear war has never happened so far .

The primary difference between us and apes is that our brains are a little bit bigger and there's probably some extra dedicated hardware for language. A little bit of extra intelligence goes a long way - we rule the world and apes live or die at our pleasure.

Computers can be immensely large and powerful, a million times the mass and power consumption of our 20-watt brains. Logically, advanced computers should end up being immensely smarter than we are, the gap would be much further than that between apes and men. While there may be some upper limit to how intelligent one can be, it is very unlikely that our brains are near it. They have to be small enough to fit through a pregnant mother's legs after all.

We have never created any intelligent being other than humans before. There are surely various unknown challenges with doing something like this for the first time. The psychology of a computer would be extremely alien, it is unfamiliar to our mode of thinking. It's interests may not be aligned with ours and its power would be very great.

This is a major threat, more important than climate change or systemic _____ism. Climate change cannot plot against us and all problems with human minds can be solved with other human minds.

/images/16809319464807184.webp

I wouldn't call a brain three times as large 'a little bit bigger.'

length vs volume. something can have way more volume despite not appearing much bigger, i am guesing that is what he means

Depends on which other apes you compare against. The difference between homo erectus (discoverer of "bang the right two rocks together the right way and you can make one a little sharper") and homo sapiens (discoverer of "bang the right two rocks together the right way and you can instantly incinerate an entire city under a radioactive mushroom cloud") is more like 1.5x. With Koomey's Law slowing down that's about a year and a half of advancement.

Why the fuck would I do the latter? I don't even have any kids so my caring about the future is pretty fucking limited.

I'm adding +1 to my dataset of "People without kids shouldn't be allowed to vote".

I don't think there's really much more to say on that but on the other hand I don't wanna get canned for low effort posting, so here's (closer to) the aforementioned "10 paragraphs when 1 would do":

If you don't have any higher / longer term aspirations than jacking off then I guess you shouldn't care about AI doomerism, but also you shouldn't care about anything medium to long term, so complaining at AI doomers for being uniquely unconvincing is kind of an isolated demand for rigor.

However, even a degenerate coomer might have some investment in the future because with voice cloning you can already make your one 11/10 porn audios. Sadly the good Elevenlabs utilities went back behind the paywall, but I'm hopeful they'll be leaked / opensourced soon, leading to a golden age of jacking off. Wouldn't want to have that shining future snatched away from you because of Skynet.

I don't wanna get canned for low effort posting

If I do not get canned for it then you should not either.

If you don't have any higher / longer term aspirations than jacking off

But I do! I have all kinds of them. That does not mean that I want to listen to 4 hour long AI danger podcasts though, or read boring essays on the topic. Surely someone can just succinctly present the matter in a few paragraphs. I am not so easily bored by verbosity! I have read The Iliad and Moby Dick several times each. I have studied advanced mathematics for fun. But I am usually bored by the typical AI doomer post.

My reference to jerking off was just a humorous example. I am in bed right now and I am probably not about to jerk off. But I also am not about to read some boring 40000 word essay about AI risk any more than I am going to go read some white paper about the threat of climate change.

It is not that I do not believe that those dangers are real. Actually, I believe that to some extent both the danger of climate change and the danger of AI are real.

It is just that I do not give a shit about either on the emotional level!

Those would change the masses at large's opinions about these topics would be wise to pay attention to the emotional level.

No, I am not going to listen to some 4 hour podcast about AI risk. Nor would I spend any more than about ten or twenty minutes at most per month at most reading about the threat of climate change. I just don't give that much of a shit! If you want to reach people who are like me in this regard... and no, we are not just degenerate coomers... we are actually probably the overwhelming majority of the human population... then please explain it in a way that nails it through my mind so that I have no choice but to agree. 4 hour podcasts or verboseshit essays won't do the trick for me or people like me. You have to make people like me actually give a shit. Right now, I don't. The problem for those who do give a shit is... people like me who don't give a shit are most of the world's population.

4 hour long podcasts and giant essays are not going to shift the needle. I am an intellectual and they have not shifted it even for me. They will shift the needle even less for people who are not intellectuals, and that is most of the world's population.

In the time and effort you spent writing your replies , you probably could have listened to half of it at 2x speed, but I agree 4 hours is way too long to make a case. People don't have time for that. But it's like watching a football game. Why watch the whole game, which takes hours, when you can just get the score online or updates on your phone/ Because the very act of watching is part of the entertainment.

Surely someone can just succinctly present the matter in a few paragraphs.

You and everyone you know and love could possibly die to this thing, it's much more likely than many think and even if it doesn't kill us all it's going to almost certainly profoundly transform nearly every aspect of everyone's lives. NVIDIA thinks the hardware necessary to run AIs is going super Moore's law at 275x per 2 years. They're already better than a lot of low level white color employees. If this thing can go super human there is a real risk the improvement goes recursive which is the generally believed most likely catalyst for a singularity type event. If this isn't enough for you then just jack off in ignorant bliss for the few remaining years before reality asserts itself. If your threat response is so atrophied and your intellectual curiosity is so empty that you can't give a single evening to figuring this out then why should anyone care about evangelizing you?

They're already better than a lot of low level white color employees.

S-tier Freudian slip.

Hah I thought that was intentional, if not agreed!

You and everyone you know and love could possibly die to this thing

Another one? That's what they said about COVID-19.

NVIDIA thinks the hardware necessary to run AIs is going super Moore's law at 275x per 2 years.

When you see exponential, think logistic.

I wish there was some way to make a counter bet . You cannot even make bets against these companies given they are private. There are betting markets but part of the problem is there is still no agreement about what it means for an AI to be aware, sentient, etc. or what an AI disaster or catastrophe would entail short of the world coming to an end. But my on take is, these fears are way overblown. The math or science of AI is real, but the doomsday argument seems like pseudoscience.

Here is how I would thwart AGI and future GPTs that does not involve regulation or moratorium : create a competing company and offer much more lucrative salaries to attract AI researchers away from Open AI and elsewhere, and then have them produce something which is harmless or useless. Although this does not address the possible threat of China.

How can this technology not make money? I saw one guy who talked with GPT-3 for a bit and got it to make a game. Idea, code and UI all made by machine, albeit human selected from many options. Sumplete, it's a bit like sudoku. Not a bad game!

The growth curve of users on ChatGPT (all but obsolete nowadays with GPT-4) is just a straight vertical line. What could be more profitable than that? There's clearly a huge market to be tapped. And your investments are all AI companies! META, Tesla, TQQQ, MSFT. Facebook gets money from ads, using its algorithm (AI) and they have a big AI research department. Tesla and self-driving, Microsoft and GPT... How can anyone start a company that competes with the biggest companies in the world for talent and get them to do useless work?

/images/16809327288607266.webp

The growth curve of users on ChatGPT (all but obsolete nowadays with GPT-4) is just a straight vertical line. What could be more profitable than that?

It's hard to turn users and growth into profits. Facebook, Alphabet do that easily.

And your investments are all AI companies! META, Tesla, TQQQ, MSFT. Facebook gets money from ads, using its algorithm (AI) and they have a big AI research department. Tesla and self-driving, Microsoft and GPT.

Humans use air, yet there is no way to make money from that. Many websites use php, yet you cannot really invest in php or C++. Just because companies use "X" does not mean profiting from "X" is possible. "Using AI" does not imply "Open AI is a good investment"

Open AI got a lot of users in part because of media hype and the WWW being much bigger today compared to a decade+ ago with the other sites, and also partnership with MSFT. The source below says 13 million daily users, which is a small fraction of the alleged 100 million users. It's possible this 100 million is inflated by lumping all Microsoft users, like outlook or windows, with Open Ai.

https://www.businesstoday.in/technology/news/story/open-ais-chatgpt-hits-100-million-users-makes-history-as-fastest-growing-app-368753-2023-02-03

By comparison, facebook had 350+ million daily users even as far back as early 2011

https://www.statista.com/graphic/1/346167/facebook-global-dau.jpg

It's hard to turn users and growth into profits.

Give it time. ChatGPT came out 5 months ago and people are already doing pretty cool things with it. Google took 3 years till it turned a profit. Facebook took 5.

Humans use air, yet there is no way to make money from that. Many websites use php, yet you cannot really invest in php or C++.

Air is non-rivalrous and non-excludable, it and PHP is in a totally different reference class to this kind of AI service. It clearly is rivalrous and excludable. AI is not a public good.

It's possible this 100 million is inflated by lumping all Microsoft users, like outlook or windows, with Open Ai.

That's false. I made an account specifically for OpenAI to use ChatGPT. Plus there are more than 100 million Windows and Microsoft users.

I’m guessing they’re counting everyone who’s used Bing for search since incorporating GPT, not just people who’ve chatted with it.

That's false. I made an account specifically for OpenAI to use ChatGPT. Plus there are more than 100 million Windows and Microsoft users.

I was just speculating on how they got 100 million. Making an account to test the service is not the same as returning to it, like in the case of Facebook or Instagram, in which people are addicted to the dopamine of social media and keep coming back. The daily user count is 13 million, is decent but still a long way from even 2011 Facebook. People are not going to pay money for something they use infrequently...some will but not enough to make this as valuable as Meta.

I am trying to think how I could use Chat GPT in my everyday life. Maybe as a research assistant?

Give it time. ChatGPT came out 5 months ago and people are already doing pretty cool things with it. Google took 3 years till it turned a profit. Facebook took 5.

As of Jan 2023, Open Ai is valued at $30 billion, maybe even more now . So a lot of growth and earnings is already priced in, hence why I would be keen on betting against it now. Even Facebook in 2009 was valued at only $6.5 billion yet it had more users and earnings compared to Open Ai.

There is a standard simple way to bet against doomsday. You tell Eliezer that you'll give him $X today, and he'll agree that if he's still alive in say, 2040 that he'll give you back $10X.

He would never accept such a bet. He strongly believes there is a very high likelihood of AI destroying the world. If he's right, he stands to gain no upside from such a bet, and I obviously would not have to honor it either.

Well the idea is that he might still have some use for that money in what he estimates to be his few remaining years.

I listened to that one, and I really think that Eliezer needs to develop a politician's ability to take an arbitrary question and turn it into an opportunity to talk about what he really wanted to talk about. He's taking these interviews as genuine conversations you'd have with an actual person, instead of having a plan about what things he wants to cover for the type of audience listening to him in that particular moment. While this conversation was better than the one with Lex, he still didn't lay out the AI safety argument, which is:

"Consistent Agents are Utilitarian + Orthogonality Thesis + Instrumental Convergence + Difficulty of Specifying Human Goals + Mesa-Optimizers Exist = DOOM"

He should be hitting those 5 points on every single podcast, because those are the actual load-bearing arguments that convince smart people, so far he's basically just repeating doom predictions and letting the interviewers ask whatever they like.

Incidentally while we're talking of AI, over the past week I finally found an argument (that I inferred myself from interactions with chatGPT, then later found Yann Lecun making a similar one) that convinced me that the entire class of auto-regressive LLMs like the GPT series are much less dangerous than I thought, and basically have a very slim chance of getting to true human-level. And I've been measurably happier since finding an actual technical argument for why we won't all die in the next 5 years.

I find it odd that Lecun writes:

Performance is amazing ... but ... they make stupid mistakes

Factual errors, logical errors, inconsistency, limited reasoning, toxicity...

Toxicity is not a stupid mistake, it is a style of communication. It is possible to convey accurate information about reality in a toxic way. It is also possible to convey total nonsense in a very polite and well-mannered way.

He doesn't seem to have an intuitive understanding of what the everyman on the street is missing. That's probably why he never starts with a ground-up outline of the whole problem. Its hard to know what fundamental assumptions your audience is missing before you get to Q&A.

I finally found an argument (that I inferred myself from interactions with chatGPT, then later found Yann Lecun making a similar one) that convinced me that the entire class of auto-regressive LLMs like the GPT series are much less dangerous than I thought

I agree that we probably don't get doom from literally just stacking more layers. The concerning thing is that these seem like easier problems to solve than I would have expected getting computers to understand concepts would be. I still think we are at least one or two BIG breakthroughs from true AGI (the kind that makes humans obsolete). Those could come tomorrow, or they could come 10 years from now, or never.

I agree that we probably don't get doom from literally just stacking more layers. The concerning thing is that these seem like easier problems to solve than I would have expected getting computers to understand concepts would be. I still think we are at least one or two BIG breakthroughs from true AGI (the kind that makes humans obsolete). Those could come tomorrow, or they could come 10 years from now, or never.

This is the way I think about it, too. Despite popular fears, Deep Blue was never going to reach AGI no matter how much you refined its algorithm or how much computing power you gave it. Chess, like writing a five paragraph essay, was a test we used to measure the unobserved variable of intelligence, so people assumed that once a machine could play chess as well as humans (or research and write a five paragraph essay as well as humans), humans were presently donezo. But that unobserved variable is a tricky thing to define, and so, to engineer.

Every year offers a small chance that someone grasps the key insight that yields an AI capable of recursive self-improvement. But it's unpredictable when this insight will arrive. We don't know the shape of the thing we're fumbling for.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats