This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
This is flogging the FTX crash horse, which if not expired yet is certainly not in the best of health, but I'm currently reading the Chapter 11 declaration by the guy put in charge of putting Humpty Dumpty back together again, and it is prime entertainment.
He is not impressed with how FTX and its web of companies was run, and he makes no bones about it. The recurring refrain all through is "However, because this balance sheet was produced while the Debtors were controlled by Mr. Bankman-Fried, I do not have confidence in it, and the information therein may not be correct as of the date stated" for all the balance sheets he's quoting. He was the guy put in to handle Enron when it was wound up, and he says (reading between the lines and you don't need to do much of that) that the FTX mess is even worse than that:
He throws shade everywhere:
(Translation: Bankman-Fried is a lying liar)
Ouch. As if Zuckerberg didn't have enough problems with the Metaverse already. Is this really the kind of PR he wants associated with it? 😁
What really interested me in all this, though, was the interview/transcript of a Twitter conversation with Bankman-Fried that Kelsey Piper published in Vox the other day. I have no idea what Bankman-Fried is trying to achieve here, but it's pretty plain that he is in a state of denial and is not accepting any responsibility for the eventual outcome. He admits he fucked up, but then shifts into blaming others, including his co-founders, and everyone who advised him to file for bankruptcy. Reading Ray's declaration, it sounds less like "I was advised" and more like "I was told do this or else", but whatever; now he is spinning a story (and I don't know if he believes this himself or was just trying it out on Piper) that if he had toughed it out and refused to file for bankruptcy he would have been able to cover most of the debts and settle up within a month or two:
Considering, according to the filing, that amongst the lawyers he consulted about that, one of them was his dad - ouch again. Sorry Dad, Sonny-boy is lumping you in with the bad advisors who led him astray. But he is in a state of delusion that he could have fixed this, or can fix it. He still can't admit he messed up because he was too greedy and not as smart as he thought he was, and all that rationalist woo about risk and utility maximisation was only a cover for bad decisions and fraud.
I would definitely recommend reading this document to get a picture of what was going on. There is no way, unless he's trying to set up for an insanity plea or operating under impairment due to drugs/mental health problems, that Bankman-Fried can deny it was all down to him. He pretty much owned or controlled every entity that was going on, it was him and literally about three others who made all the decisions, and they seem to have treated the interlocking parts as their own private piggy-bank (e.g. "three loans by Alameda Research Ltd.: one to Mr. Bankman-Fried, of $1 billion; one to Mr. Singh, of $543 million; and one to Ryan Salame, of $55 million"). Then read the Vox article to see how he is admitting all his EA/altruism talk was basically telling them what they wanted to hear so he'd be popular and well-liked and they'd trust him, because getting people to like you is winning and winning is all that counts.
And this set-up was having billions of dollars in investment funding thrown at it, and it was less well-organised than a school bake sale when it came to handling and keeping track of what money was coming in and where it was going.
I have a question. Why, after this fiasco with FTX, should I have any faith that the Effective Altruism movement has any handle on existential risk or any capability to determine what actions will increase or decrease said risk? My impression is that this management of existential risk is a substantial part of EA's brand. Especially William MaCaskill and longtermism as a movement. Some of the leading lights of the EA movement (like MaCaskill) were apparently unable to manage the well defined risk of "maybe this guy running a cryptocurrency exchange is a scam artist" but I'm supposed to believe they have a handle on the vastly more nebulous and ill defined risk of "maybe an unfriendly artificial intelligence extincts humanity." Why should I believe this?
First of all ... EA doesn't claim to 'have a handle on AI risk'. EA claims to be worried about AI risk, and not have a handle on it, hence the worry.
from @moskov on twitter: "The other day a very close friend said "of course I believe pandemic and AI risk are huge problems, but you've got them covered" and I screamed "NO WE ARE FAILING". That wasn't even about money per se, just making any kind of attempt."
Second, why do you need to have 'faith in EA'? You should evaluate their actions individually. Entirely possible for GiveWell to be good but their AI risk work to be bad, even if FTX didn't go down.
More options
Context Copy link
Existential risk is a hack job for human psychology. Once you can convince someone (especially yourself) that something is an existential risk to all life, why you can do goddamned anything you can convince yourself has even a tiny chance of preventing it. It's "Deus Vult" for atheists. The further into the future this dystopia lies, the less likely you'll ever have to deal with the fallout of your failed predictions. We've been on "ten years to save the world from global warming" for a good forty years now.
My stupid moron religious parents think the world is going to end because people are sinful and God is vengeful. Really smart people believe the world is going to end because.......people are sinful and (AI) gods are vengeful. Environmentalists believe the world is going to end because......people are sinful and mother earth is vengeful.
FWIW, I think AI x-risk is real, and this doesn't match my psychology. I am unhappy about this, and wish I could just go back to thinking about purely technical problems. I dislike the flavor of AI risk work (qualitative research, politics, and activism) and also dislike being seen as a self-important, deluded ninny (which others are right to suspect of me).
More options
Context Copy link
None of that makes their claims wrong. Is this even a psychology thing? Things like 'existential risks to all life' do exist (for historical examples: atmosphere oxygenation "caused the extinction of many existing anaerobic species on Earth ... constituted a mass extinction", Chicxulub "a mass extinction of 75% of plant and animal species on Earth, including all non-avian dinosaurs.", ice ages, human population bottleneck), and technology makes it easy for us to do similarly bad things. Preventing such things is important, and it would be stupid to write them off. So this isn't even a psycholoical "hack", it's just ... something important that one can be wrong about. But it's not any more worth ignoring than it is to ignore the doctor saying you need surgery or you'll die, because faith healers say you need to get a protection spell or you'll die. That X-risk leads people to make difficult and complex decisions is good
Yes, all predictions of the apocalypse up until this very second have been wrong.
Yes, the next one might be right.
No, it won't be, but it might.
Sooner or later, it's either the big 'ol HDU or something else.
There is an absolutely unbroken line of apocalyptic dreamers who prefer their fantasy to dealing with the reality of the world. It goes back to the beginning of time when the first dipshit looked up at a shooting star and said "that means the gods are mad at us, we're all gonna die".
I'm doing a rationalist calculation. What are the odds that I meet the first Jeremiah in the history of the world to be right? I mean, a lot of people have lived a lot of lives and most of them thought the world was going to end in their lifetime. I'm pretty hot shit, but I doubt I'm lucky enough to even be alive at the same time as that guy, much less live in the same country, speak the same language, share enough personality quirks with to wind up in the same internet forums etc. What are the odds?
Pretty goddamned good, since I've lived through fourteen to twenty apocalypses (apocalypsi? apocalyps'?) to date. What a life. Fire, flood, the return of Christ, acid rain, Y2K, nuclear war (several times!), the Macarena, Global Warming and now AI? Bring it on, I say.
I gave several examples of literal apocalypses though
Again, it's entirely possible for massive technological change to make apocalypses possible. There clearly is a difference between 'god will fire lighting boom everyone for not being religious enough' and 'the billions of dollars and millions of man hours of the smartest people on the world are being invested in AI, what if it works'.
Given that EA spends more time and money on malaria nets than AI risk, this is clearly not an accurate statement about them. More generally, that doesn't actually make AI risk false.
Even if EA and lesswrong were - entirely - irrational and religious cults around AI risk, that wouldn't make AI risk false. And there are stupid, illogical cults around AI - there was and still is a lot of popular scifi larp about "the singularity", "mind uploading", etc. This doesn't make the AI go away.
Yeah, but those occurred on the rough timescale of once every billion years, and all prior to anything anywhere near humanity existing. The ratios on apocalypse-level events Humans have worried about to things that have actually happened during a timeframe that concerns us is therefore at least that high.
Technology has increased the rate of change of everything, though. a million years of hunter gathering, 100k years of fire, 10k years of agriculture, civilization ... 300 years of industry, 60 years of computers... If AI doesn't happen, what does happen in 10k years, and why hasn't AI taken power from humans yet?
More options
Context Copy link
More options
Context Copy link
I got raised on "we must avoid nuclear war, if it ever happens it will doom humanity, never mind the hundreds of millions killed by the bombs, the nuclear winter afterwards will mean we all starve and freeze to death in the dark". Nuclear war was the existential risk of the 60s-80s. This animated film frightened the life out of a generation of kids.
Now I'm reading "eh, nuclear war isn't that bad, sure it will kill a lot of people but not everyone, it will not be the end of civilisation much less humanity, and even nuclear winter was over-exaggerated".
What changed about nuclear bombs in the meantime? Nothing, so far as I can see, but the attitude around that risk has changed. Now it's climate change that is the existential risk of our time. AI can just take its place in the queue of "This time for sure, says Chicken Little" about the sky falling.
Nuclear risk is actually one of the EA Big Four - AI, Bio risk, Climate change, and Nuclear war. I have met far more people in EA that actually care about nuclear war than anywhere else.
The reason AI is getting more salience is because it's perceived as more neglected which is one of the key criteria of an EA cause area. Nuclear war is mainstream and nobody wants it, so it isn't as neglected as the other areas.
More options
Context Copy link
I'd like to note, though, that when I asked this question from the perspective you've mentioned, I got at least one reply to the effect of "you could still be boned." Nuclear war might not end all human civilization immediately, but even post-Cold-War media still tended to portray the post-nuclear world as pretty bleak even if there were still some people alive (after all, in such a world, the living might envy the dead). That is to say, even today, nuclear war could get pretty bad, we just have reasons/copium to believe the possibility space also contains scenarios that aren't "rubble and deserts everywhere."
More options
Context Copy link
Your argument goes like 'society was wrong and people lied about nuclear war, and used that to manipulate people - so that must be what's happening now'. Which ... sure, that can happen - it constantly happens - but the reason people even have a desire to avoid x-risks is because it is important to avoid disasters when disasters exist, and disasters sometimes exist.
This is like saying - "you're worried about high crime? people in the 1900s were worried about racialized crime destroying society, and they were wrong, and racist. Therefore crime doesn't matter". You can't write off the entire idea of 'bad thing happening' because people misuse the idea!
What I'm saying is, this is not my first rodeo. If all the x-risks that were sure-fire guaranteed gonna happen as prognosticated over the course of my life had even one of them happen, we'd be disposed of by now.
So "Oh no AI is gonna doom us unless!" talk is nothing special. "But AI is different" - yeah, well, so were nuclear weapons. It's not AI that is the risk, it's humans. We are the greatest threat to ourselves.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I did too. They all were considered possible, likely or certain by millions of geniuses in their day (except the silly ones, of course). They all had a reason why this time it was for real. They all happened, for some definition of "happen", and they all did not result in the end of the world, humanity, life or anything else so dire. I'm sure AI is dangerous. I'm sure we'll have some colossal fuckups with it that will probably damage something important. When this happens, the frenzy will begin in earnest. Timelines will be settled on, politics will change, a solution will be found, and we will learn to live with it, as we have with nuclear weapons.
Whichever generation of asshole eschatologists alive at the time will write a million books saying they averted the apocalypse. Ten seconds later, it will be something else, and everyone will forget about it. The End of the World is dead, long live the End of the World.
Maybe I'm wrong. Tell you what, if the world ends due to AI, I'll give you a million dollars.
Have you read any yud or lesswrong writing on AI safety? They put a lot of effort into addressing the exact concerns you've laid out, in a way that you don't seem to acknowledge. But leaving that - why though? Why will the competent people find a solution? It's clear how we are able to find a solution to, e.g. nuclear weapons, religious conflict, etc. But AI will be - it's argued - not a simple mechanism we can intelligence and coordinate around, but smart on its own. As a random example - what is the political solution to "AIs now control the global economy"? The AIs are going to be the one "finding solutions", not "human politicians". You can't psychologize your way around a gun to your head, and no amount of "you're just scared its ok the grownups will handle it" will physically prevent the complexity of AI!
Yes, yes, very smart people disagree with me. Your argument is to handwave me at the vast canon of AI scribbling? I've read the big ones, and they are as unconvincing as they are hysterical. It's a very specific style, one I recognize well from my upbringing in a millennial faith-healing cult. It all sounds very convincing, if you haven't been down this road before.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Well, no one else is taking the issues seriously.
Stuart Russell wrote a whole book.
And what came of it?
-Marx
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The answer seems obvious: instead of taking someone's word for or against ai xrisk being a thing, the arguments for it have to be evaluated on their merits, and you can decide for yourself whether it is something to be concerned about based on that.
More options
Context Copy link
My personal guidance has long been that faith in individuals (and small groups) is almost always misplaced. Faith in principles can do pretty well, but IRL humans are quite fallible and rarely live up to our expectations. There is no living person whose word I would treat as sacrosanct without a willingness to do my own research.
IMO nobody has satisfactorily explained why AI risk outweighs, say, the existential risk of an extinction event by anthropogenic (nuclear war or catastrophic ecological disaster) or other (asteroid, supervolcano, nearby supernova). I don't think we have good handles on the relative magnitudes of risk on these. At some point you're really just acting on your priors.
Many people within EA are also not convinced that AI risk outweighs all other risks. This is more of a strawman that gets heaped on EA because the most visible EA detractors latch onto the wackiest part of the movement.
More options
Context Copy link
Humans 'took over the planet' and displaced billions of other animals just by being smarter. Why wouldn't AI agents do the same? Our intelligence relates to physical mechanisms somehow, and computers are a much faster and more efficient way of implementing similar things, so why wouldn't computers be able to do so better? Look how rapidly technology is integrated into human life and the economy, won't that just continue to accelerate as it has over the past 300 years? What does a human do, what happens to the economy, when complex AI interactions drive more and more of it?
There's some debate on that, there's a competing theory that we "took over the planet" because we had language and could thus learn from events that we ourselves did not witness.
Transformers and large neural nets generally are universal ML system that can do moderately-intelligent tasks like create images, play games, optimize all sorts of real-world tasks, and also understand language. Human intelligence is similar - use and understanding of language depends and is made of general intelligence. If dogs could talk, they wouldn't be smarter. And - they can talk, sort of, they have verbal and body-language signals that mean things, they just can't do larger-scale things with that.
This feels like affirming the consequent/assuming the conclusion.
If dogs could talk would they need to be smarter? How much rougher would the lions' life be if the Buffalos and Gazelles could collude?
yes, dogs can't play sports (even if they had the right physical form factor) or compose music as well as humans despite neither requiring language. buffalos can communicate, and do, the ability to understand complex communications wold require intelligence, and even if they had that minimal intelligence that doesn't get them to toolmaking, let alone physics or military planning. is this a bit?
No, because it seems equally clear that intelligence doesn't get you that far on it's own.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I feel like we actually DO have a pretty good handle on these existential risks. Definitely we can model the likelihood of extinction level asteroids and supernovas. The base rate here is very low, and we're improving our odds by tracking asteroids. Supervolcanoes are not extinction level events.
Nuclear war or ecological disaster seem unlikely to be extinction level events. Nuclear weapons are just not powerful enough, although they could potentially reduce the population by a lot. Global warming is centuries away from being an existential concern.
Rogue AI on the other hand has the potential to kill all humans. We can argue about whether the chances are 1% or 99%. And we can argue about whether it will happen in 10 years or 50 years. But if we're drafting talent in the NBA draft of extinction events, AI gets drafted #1, and it's not even close. It combines high likelihood in the near-term with the potential to kill ALL humans, not just 5% of them or whatever.
Yeah, but that's not the Drowning Child, is it? They charity-mugged everyone with that nugget from Singer: if you saw a child drowning right now, in front of your very own eyes, wouldn't you save it? What kind of heartless monster would not? And so if you accept that you should save the drowning child, then you must accept that you should save the children who will die without malaria nets.
I know the malaria nets projects have now got more money than they know what to do with, so it's not hypocritical for EA to move to a new good cause. But it is hypocritical to move to a cause where the advice now is "Forget about that drowning child, if you jump into the pond you might catch a cold and be home from work sick and then your contribution to stopping AI from paperclipping us all will be lost for that valuable period of a week! Sure, we don't know if the AI is coming in ten or fifty years, and the kid is drowning right now, but imagine all the kids in fifty years time instead!"
More options
Context Copy link
There are people who also argue that it will happen "never". Anyway I'd say this is a threat in similar ballpark as "being invaded by alien species". What is a chance that humanity's technological progress woke up some Von Neumann probe from some ancient civilization somewhere in Oort cloud which is now ramping up self-replication capacities and building up an invasion spacefleet set to destroy Earth? Existence of such probes throughout the galaxy could definitely be one answer for Fermi paradox, whatever the chance you assign to such a possibility multiply it with existential threat and you are Pascal mugged toward giving it a lot of thought.
But similarly to the AI problem, it is hard to conceive what to do about it. Maybe not search for ancient civilization by Active SETI and maybe increase manufacturing of nuclear rockets capable of targeting things on our orbit?
Going back to AI, I have yet to see any material results of the supposed "friendly AI" research. We do not even know what technology will lead to such an AI which means it may be hard or even impossible to imbue it with certain morality. Not to even talk if EA people and affiliated researchers are the best people for that job given their own morality. In fact if one is as serious as you are here about it, probably the best course of action would be to go full Terminator 2 and assassinate all leading AI researchers and bomb all the research laboratories with hope that we will eventually land upon some form of AI Inquisition type government banning all research into the area under severe penalties. Of course this looks too crazy even for AI types, but it would be logical conclusion here.
Well they're not serious thinkers, are they? Where is the law of physics that says it's impossible to make a rogue AI? Even if such a law did exist, never is a very strong word. I'd be cautious before saying we would 'never' find some way around thermodynamics, the most solid foundation we have. Who knows what could be achieved 500,000 years after the Scientific Revolution? We're only 300 or so years in, there may be a few revolutions to come.
It's outrageously silly to say 'never' when we have so many questions still unanswered, when AI is advancing at such a rapid pace.
More options
Context Copy link
I almost put aliens in my comment. The threat from extraterrestrials is similar to AI in that it is difficult to quantify but at the same time the potential damage is 100% extinction. And since, in either case, we have no base rate we have to make assumptions of likelihood from first principles. This is what some people have difficultly accepting, probably the same type who weren't worried about nuclear weapons in the 1940s.
I also agree that there is not much that can be done. Although actively trying to get aliens to find us does seem uniquely stupid. I guess I just get frustrated when people conflate "minor" threats such as climate change or supervolcanoes with things that are much more serious.
More options
Context Copy link
I mean, huge amounts of talent and capital are being poured into building AGI, and we know it's physically possible (because human brains exist). So to be sure that it'll never happen seems like a stretch.
More options
Context Copy link
More options
Context Copy link
Not really. At least we know those other things are actually within the realm of possibility. We have no reason to even believe (yet) we can invent an AI that could kill us all, let alone one that would. AI risk is just not worth taking seriously at this point in time.
It seems important to consider before you create something that could (because if you wait it might be too late).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't see why FTX should really change your views there. SBF is just one person and doesn't define the movement. Fraud existed before FTX and will exist after it.
The bigger question is why you should have any faith in the EA movement to begin with. I would argue you shouldn't. I certainly don't. Their arguments are IMO naive and the millenarian fervor certainly doesn't give them a veneer of respectability. I mean, they're patting themselves on the back for rediscovering charity. Wow, congratulations, that concept has only been around for a few thousand years. As far as I'm concerned EA is just a honeypot for self-important status-seekers trying to feel good about belonging to a cause. They're this generation's flower children.
I largely agree with your criticisms, but my impression is that most mainstream charities maintain a cultivated ignorance about their actual impact. I don't think much of our ability to define or measure the things we care about achieving, but at least EAs are pushing for a norm of trying to do so.
Yeah, if you're measuring "Did Curtains For The Congo really do any benefit for the people there with their fundraising campaign to send soft furnishings to Kinshasa?" you can certainly set up a norm of defining and measuring.
But AI is now the same kind of "we don't what effect, if any, our efforts will have or if we're even trying the right approach, but we believe passionately in this cause so we are determined to throw money, time and effort at it" case as the traditional charities they twitted for not being very effective. They may be right to be concerned, and what they are doing might be right - or the completely wrong approach and they are looking the wrong way while the real threat is coming from a different direction. We'll only find out in 10-50 years. But never mind that we have no way of defining or measuring if this is effective, think of the risk! Pour all your money and energy into it!
I agree with your framing of the difficulty of measuring progress on AI risk, and I think most EAs would as well. They would say that we should still try to measure such progress, or at least recognize that most of our efforts will probably go to waste because we can't, as you point out.
I think most EAs would say that even though we know we're going to waste most of the time, money, and energy we put into AI risk (because it's so hard to measure) it's still worthwhile. So I don't think this is an instance of hypocrisy, just an instance where plan A doesn't work.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm not sure what you're suggesting. If Sequoia and others are willing to invest hundreds of millions of their own money into FTX, why should someone receiving charity expect to be able to do better due diligence? Why should they try? "Existential Risk" specifically refers to humanity's existence, not any particular replaceable entity that might go under.
yeah this. doing diligence is hard enough with a public company. it's way, way harder with a private one
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link