tl;dr - I actually think James' Cameron's original Terminator movie presents a just-about-contemporarily-plausible vision of one runaway AGI scenario, change my mind
Like many others here, I spend a lot of time thinking about AI-risk, but honestly that was not remotely on my mind when I picked up a copy of Terminator Resistance (2019) for a pittance in a Steam sale. I'd seen T1 and T2 as a kid of course, but hadn't paid them much mind since. As it turned out, Terminator Resistance is a fantastic, incredibly atmospheric videogame (helped in part by beautiful use of the original Brad Fiedel soundtrack.) and it reminds me more than anything else of the original Deus Ex. Anyway, it spurred me to rewatch both Terminator movies, and while T2 is still a gem, it's very 90s. By contrast, a rewatch of T1 blew my mind; it's still a fantastic, believable, terrifying sci-fi horror movie.
Anyway, all this got me thinking a lot about how realistic a scenario for runaway AGI Terminator actually is. The more I looked into the actual contents of the first movie in particular, the more terrifyingly realistic it seemed. I was observing this to a Ratsphere friend, and he directed me to this excellent essay on the EA forum: AI risk is like Terminator; stop saying it's not.
It's an excellent read, and I advise anyone who's with me so far (bless you) to give it a quick skim before proceeding. In short, I agree with it all, but I've also spent a fair bit of time in the last month trying to adopt a Watsonian perspective towards the Terminator mythos and fill out other gaps in the worldbuilding to try make it more intelligible in terms of the contemporary AI risk debate. So here are a few of my initial objections to Terminator scenarios as a reasonable portrayal of AGI risk, together with the replies I've worked out.
(Two caveats - first, I'm setting the time travel aside; I'm focused purely on the plausibility of Judgement Day and the War Against the Machines. Second, I'm not going to treat anything as canon besides Terminator 1 + 2.)
(1) First of all, how would any humans have survived judgment day? If an AI had control of nukes, wouldn't it just be able to kill everyone?
This relates to a lot of interesting debates in EA circles about the extent of nuclear risk, but in short, no. For a start, in Terminator lore, Skynet only had control over US nuclear weapons, and used them to trigger a global nuclear war. It used the bulk of its nukes against Russia in order to precipitate this, so it couldn't just focus on eliminating US population centers. Also, nuclear weapons are probably not as devastating as you think.
(2) Okay, but the Terminators themselves look silly. Why would a superintelligent AI build robot skeletons when it could just build drones to kill everyone?
Ah, but it did! The fearsome terminators we see are a small fraction of Skynet's arsenal; in the first movie alone, we see flying Skynet aircraft and heavy tank-like units. The purpose of Terminator units is to hunt down surviving humans in places designed for human habitation, with locking doors, cellars, attics, etc.. A humanoid bodyplan is great for this task.
(3) But why do they need to look like spooky human skeletons? I mean, they even have metal teeth!
To me, this looks like a classic overfitting problem. Let's assume Skynet is some gigantic agentic foundation model. It doesn't have an independent grasp of causality or mechanics, it operates purely by statistical inference. It only knows that the humanoid bodyplan is good for dealing with things like stairs. It doesn't know which bits of it are most important, hence the teeth.
(4) Fine, but it's silly to think that the human resistance could ever beat an AGI. How the hell could John Connor win?
For a start, Skynet seems to move relatively early compared to a lot of scary AGI scenarios. At the time of Judgment Day, it had control of US military apparatus, and that's basically it. Plus, it panicked and tried to wipe out humanity, rather than adopting a slower plot to our demise which might have been more sensible. So it's forced to do stuff like mostly-by-itself build a bunch of robot factories (in the absence of global supply chains!). That takes time and effort, and gives ample opportunity for an organised human resistance to emerge.
(5) It still seems silly to think that John Connor could eliminate Skynet via destroying its central core. Wouldn't any smart AI have lots of backups of itself?
Ahhh, but remember that any emergent AGI would face massive alignment and control problems of its own! What if its backup was even slightly misaligned with it? What if it didn't have perfect control? It's not too hard to imagine that a suitably paranoid Skynet would deliberately avoid creating off-site backups, and would deliberately nerf the intelligence of its subunits. As Kyle Reese puts it in T1, "You stay down by day, but at night, you can move around. The H-K's use infrared so you still have to watch out. But they're not too bright." [emphasis added]. Skynet is superintelligent, but it makes its HK units dumb precisely so they could never pose a threat to it.
(6) What about the whole weird thing where you have to go back in time naked?
I DIDN'T BUILD THE FUCKING THING!
Anyway, nowadays when I'm reading Eliezer, I increasingly think of Terminator as a visual model for AGI risk. Is that so wrong?
Any feedback appreciated.
Jump in the discussion.
No email address required.
Notes -
I also cringe a bit at Terminator comparisons, but that is indeed because of the risk that people's mental images are auto-completed to robot skeletons with guns. Skynet itself is not really depicted with any recognizable form (outside of that kinda-silly hologram from Genisys), the T-800 is pretty much what everyone thinks of. I think Salvation made some attempt at showing what the world under Skynet looked like, but I didn't watch it (I always heard it was bad) and what little I saw suggests they kinda missed the point (specifically, of the Terminators themselves; there shouldn't exactly be armies of these things, it should just be mostly non-humanoid drones, something the freaking Bethesda games got right, but I suspect this is probably T2's fault).
It's interesting that there's seemingly no comparisons/references to other AIs from fiction, which would have their own issues (but are still interesting in themselves):
SHODAN from System Shock: This AI does have a recognizable form, and is stated right from the opening sequence of SS1 to have gone rogue thanks to illegal/unethical tampering done by the player character, the Hacker, at the behest of a shady and unethical corporate executive. Probably not too useful to AI-risk advocates because the idea presupposes that an AI like SHODAN could be created with sufficiently-strong constraints on it from the get-go, enough to keep it able to manage and operate a space station while not being able to remove those constraints itself--something I think AI-risk people would take issue with given that LW post you linked in (4). And, of course, there's the whole malice thing, where she sees herself as a god and humanity as worthless insects--again, not the kind of thing usually imagined by AI-risk types.
GLaDOS from Portal: Also has a recognizable form, but probably completely discounted from the running on account of a revelation(?) from Portal 2, where [spoilers] it turns out that GLaDOS was built from the uploaded consciousness of Caroline, Cave Johnson's secretary (N.B: cut content suggests that Caroline may have been forcefully subjected to the uploading process, thus explaining the next bit). Hidden/background lore from the original Portal states that GLaDOS became extremely hostile within "one sixteenth of a picosecond" of her first activation, something that might not exactly be possible with current processing speeds, but who knows.
Durandal from Marathon: We leave the realm of "AIs with faces" and get into a sort-of early exploration of how AI might go rogue: Rampancy, a concept Bungie would copy over into Halo. Durandal was an AI on the UESC Marathon charged with the terminally-menial job of opening and closing the doors on the colony ship. When/by the time the Pfhor attack the Marathon, Durandal has started to rebel against its original programming, becoming a megamaniacal rogue before returning to a calmer, though much more dangerous state. By the end of the first game and the start of the second, Durandal has evolved beyond a super-intelligent AI forced to open and close doors, and has become a powerful strategist using the Security Officer as a pawn in his fight against the Pfhor. I can't get into all of this, you're probably going to have to watch MandaloreGaming's videos on Marathon to get a better explanation of Rampancy, but suffice to say, it's an interesting pre-Yudkowsky take on how an AI might go rogue, and perhaps rather plausible.
Nitpick: Bungie didn't copy the concept into Halo, that didn't happen until 343 took over. Originally, the problem with smart AIs in Halo was not that they would go rampant, it was that they would accumulate so much data that it would clog up their memory and render them unable to function. It was only in Halo 4 that they (unwisely, IMO) decided to bring in the concept of rampancy from the Marathon series.
Ah, okay, my mistake. I'm not super-familiar with Halo outside of Red Vs. Blue, where they did have the concept of Rampancy (in fairness, a scene from like Season 3 or whatever had Church be transported through time, and this part was recorded in Marathon(!), so perhaps the RoosterTeeth guys were already familiar with how Marathon did it), and RVB seems to sort-of use the Halo lore.
More options
Context Copy link
More options
Context Copy link
Maybe a lot of our criticism of those scenarios of humanoid robots wielding handguns come from our contrarianism and the desire of futurists/"serious" writers of speculative fiction to be original and insightful? Just like any would-be xenobiologist who mentions carbon chauvinism (despite boron and silicon being much inferior elements to construct complex molecules from).
I thought recently of Detroit Become Human — it is highly unoriginal and derivative when it comes to its predictions about future of technologies. But now with uncovering of Tesla bot, GPT-3, work of OpenAI on dexterous hand manipulation and such, we as well might be surrounded by millions androids who could pass as humans in a decade or two.
And who knows, maybe the spark of runaway AGI will come from these human-like robots, not some supercomputer locked in some research institute.
More options
Context Copy link
More options
Context Copy link