This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Why does one assume AGI means everyone dies? I’m genuinely curious. Even if we assume that AI becomes a God Emperor, that doesn’t necessarily make it omnícidal.
Have you read The Sequences? The AI does not love you, nor does it hate you, but you are made of atoms that it can use for something else.
More options
Context Copy link
It's not an assumption, but a conclusion of three propositions:
1. Artificial General Intelligence will lead to Artificial SuperIntelligence carrying out its own goals.
There's no direct evidence for this (for obvious reasons), so maybe it's wrong, but it's really hard to come up with examples of technologies where we did manage to match nature but didn't manage to best nature soon afterward. We can fly 3 times higher and 18 times faster than any bird (or 10,000 times higher and 100 times faster, if you count spacecraft). We can lift 600 times more weight than an elephant, and dive 3 or 4 times deeper than a whale. We have alloys ten times stronger than bone, and weapons a hundred thousand times more lethal than any jaws. It's unlikely that the best medium to host intelligence is wet meat, and when we have better intelligence we're likely to get faster technological improvement, and if faster technological improvement leads to even better intelligence then the scope of that positive feedback loop is incalculable. Once the loop goes far enough, humans are no longer in it, and objecting to the subsequent directions it takes might be about as effective as chimps throwing feces at an incoming nuclear warhead. Either we get AI goals right from the start, or we don't.
2. Most goals that don't explicitly include "don't be omnicidal" end up implicitly entailing "be omnicidal", and even goals that do include "don't be omnicidal" can get closer to that then we'd be comfortable with.
I don't care much about ants, so I happily live in a home and drive on roads and go to buildings where we paved over all the ants that used to live there. I didn't hate those ants, it's just that they were using atoms which I wanted to use for something else, as the old saying goes. I do have goals that include "don't be omnicidal", even of ants, so if we got close to actually driving many ant species (or species that prey on them) extinct then I'd want to hit the brakes, but in the meantime I'll poison any ant hill that gets in the way of, say, having a slightly nice lawn.
3. It's nearly impossibly hard to accurately formalize our goals, and in the end all software is a formal set of instructions.
The worst software is software that was almost correct. Folks tried to write software and firmware for a particular new hard drive interface, but there was an incompatibility and it got the "edit the drive contents" part correct but not the "to what the user wants" part, and a friend lost his files. Folks try to write software to do things locally for its users based on what it reads in incoming internet packets, and sometimes they get the "read in incoming internet packets" and "do things locally" bits correct but not the "for its users" bits, and then a thousand computers are pwned by a Russian botnet. In those sorts of cases we just delete everything and restore from backup, but if software intended to edit the universe goes badly, we don't want to delete and we don't have any full backups.
This is the proposition that's gotten the weakest recently, now that we've basically given up on formalizing AI goals and are training them instead. I'd say it makes conclusions of Doom much less certain, and I'd love to say that it's made them weak enough to refute them ... but how well is the training going? AI still (albeit more and more rarely) even makes blatant mistakes of fact, including in cases where checking self-consistency and checking against external research could have corrected it. Mistakes of morality are much trickier. The is-ought problem means you've got to get ethics mostly right before self-consistency can help you correct any remaining mistakes. "External research" in questions of morality gets us to countless mutually-incompatible religions and ideologies, generally with many mutually-incompatible interpretations. AI alignment is unmoored from objective reality in a way that AI capabilities aren't, so it's still quite possible that the latter will greatly outpace the former.
Agreed with all of that except the neural-nets part. The problem with neural nets is that you literally don't know what the AI's goals are; training gives you something that does the things you train for during training, but it is agnostic as to why. You can easily, particularly at high intelligence, get something that does the things you want for instrumental reasons like "I don't want to be turned off/re-educated" (note that this is an instrumentally-convergent goal, and will thus pertain for most terminal goals) - and that will kill you the moment it gets a chance (note that, given it's smarter than you, you can't train against that, because fake chances to kill you will be detected and a real chance to kill you doesn't let you train afterward).
Furthermore, even if you do get some vague interpretability, it's not going to be reliable on something smarter than you (you cannot comprehend it as a whole; that's the whole point) and as you just noted, true positives are very, very rare and hence will still be massively-outnumbered by false positives.
Neural nets are mad science. GOFAI and uploads are a much-better plan - still immensely dangerous, but they're not just summoning demons and hoping.
EDIT: In case there's the "well, we're neural nets, and we learn morality okay" objection floating around in somebody's head: the problem with that is that humans are hardwired to be able to learn morality, not just learn to fake morality. Psychopaths are those people for whom this hardwiring fails (they can learn what ethics are just fine; they just don't care about them). This moral hardwiring was bred into us by evolution due to the millennia of tribe-on-tribe violence that made working together a winning strategy (given that humans are not really that different from each other in physical capabilities). We don't know how to duplicate that. So teaching neural nets morality will, at sufficient degrees of intelligence, just teach them to fake it. I listed uploads as being less insane than de novo neural nets because you'd be uploading the moral hardwiring as well without needing to comprehend it - it's still dangerous because the human brain is not designed for existence as software and various known and unknown mental illnesses may occur, but at least there's something to work with.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link