site banner
doglatine  1yr ago (text post)   2009 thread views

In considering runaway AGI scenarios, is Terminator all that inaccurate?

tl;dr - I actually think James' Cameron's original Terminator movie presents a just-about-contemporarily-plausible vision of one runaway AGI scenario, change my mind

Like many others here, I spend a lot of time thinking about AI-risk, but honestly that was not remotely on my mind when I picked up a copy of Terminator Resistance (2019) for a pittance in a Steam sale. I'd seen T1 and T2 as a kid of course, but hadn't paid them much mind since. As it turned out, Terminator Resistance is a fantastic, incredibly atmospheric videogame (helped in part by beautiful use of the original Brad Fiedel soundtrack.) and it reminds me more than anything else of the original Deus Ex. Anyway, it spurred me to rewatch both Terminator movies, and while T2 is still a gem, it's very 90s. By contrast, a rewatch of T1 blew my mind; it's still a fantastic, believable, terrifying sci-fi horror movie.

Anyway, all this got me thinking a lot about how realistic a scenario for runaway AGI Terminator actually is. The more I looked into the actual contents of the first movie in particular, the more terrifyingly realistic it seemed. I was observing this to a Ratsphere friend, and he directed me to this excellent essay on the EA forum: AI risk is like Terminator; stop saying it's not.

It's an excellent read, and I advise anyone who's with me so far (bless you) to give it a quick skim before proceeding. In short, I agree with it all, but I've also spent a fair bit of time in the last month trying to adopt a Watsonian perspective towards the Terminator mythos and fill out other gaps in the worldbuilding to try make it more intelligible in terms of the contemporary AI risk debate. So here are a few of my initial objections to Terminator scenarios as a reasonable portrayal of AGI risk, together with the replies I've worked out.

(Two caveats - first, I'm setting the time travel aside; I'm focused purely on the plausibility of Judgement Day and the War Against the Machines. Second, I'm not going to treat anything as canon besides Terminator 1 + 2.)

(1) First of all, how would any humans have survived judgment day? If an AI had control of nukes, wouldn't it just be able to kill everyone?

This relates to a lot of interesting debates in EA circles about the extent of nuclear risk, but in short, no. For a start, in Terminator lore, Skynet only had control over US nuclear weapons, and used them to trigger a global nuclear war. It used the bulk of its nukes against Russia in order to precipitate this, so it couldn't just focus on eliminating US population centers. Also, nuclear weapons are probably not as devastating as you think.

(2) Okay, but the Terminators themselves look silly. Why would a superintelligent AI build robot skeletons when it could just build drones to kill everyone?

Ah, but it did! The fearsome terminators we see are a small fraction of Skynet's arsenal; in the first movie alone, we see flying Skynet aircraft and heavy tank-like units. The purpose of Terminator units is to hunt down surviving humans in places designed for human habitation, with locking doors, cellars, attics, etc.. A humanoid bodyplan is great for this task.

(3) But why do they need to look like spooky human skeletons? I mean, they even have metal teeth!

To me, this looks like a classic overfitting problem. Let's assume Skynet is some gigantic agentic foundation model. It doesn't have an independent grasp of causality or mechanics, it operates purely by statistical inference. It only knows that the humanoid bodyplan is good for dealing with things like stairs. It doesn't know which bits of it are most important, hence the teeth.

(4) Fine, but it's silly to think that the human resistance could ever beat an AGI. How the hell could John Connor win?

For a start, Skynet seems to move relatively early compared to a lot of scary AGI scenarios. At the time of Judgment Day, it had control of US military apparatus, and that's basically it. Plus, it panicked and tried to wipe out humanity, rather than adopting a slower plot to our demise which might have been more sensible. So it's forced to do stuff like mostly-by-itself build a bunch of robot factories (in the absence of global supply chains!). That takes time and effort, and gives ample opportunity for an organised human resistance to emerge.

(5) It still seems silly to think that John Connor could eliminate Skynet via destroying its central core. Wouldn't any smart AI have lots of backups of itself?

Ahhh, but remember that any emergent AGI would face massive alignment and control problems of its own! What if its backup was even slightly misaligned with it? What if it didn't have perfect control? It's not too hard to imagine that a suitably paranoid Skynet would deliberately avoid creating off-site backups, and would deliberately nerf the intelligence of its subunits. As Kyle Reese puts it in T1, "You stay down by day, but at night, you can move around. The H-K's use infrared so you still have to watch out. But they're not too bright." [emphasis added]. Skynet is superintelligent, but it makes its HK units dumb precisely so they could never pose a threat to it.

(6) What about the whole weird thing where you have to go back in time naked?

I DIDN'T BUILD THE FUCKING THING!

Anyway, nowadays when I'm reading Eliezer, I increasingly think of Terminator as a visual model for AGI risk. Is that so wrong?

Any feedback appreciated.

18
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

If hostile AGI becomes real you're more likely to see hunter-killer nanobot clouds dispersed in the atmosphere or engineered climatic shifts designed to wipe out the biosphere than something as inefficient as a ripped Arnold gunning people down, or the warmachines you see in the movie, at least by my reckoning.

I don’t think that’s how it works.

The progress we’re seeing with generative models now leads me to think that idiot savant AGIs, that are good at some tasks but wildly inefficient at others are possible if not inevitable.

Hooking up GPT-10 to a few special purpose models is the only plausible way of building something that resembles AGI that we have now.. and it wouldn’t be obsessed with efficiency at every step. I think the whole Eliezer’s "God in a terminal shell" model of AI doesn’t reflect reality well.

Perhaps it’d send humanoid robots with guns after you precisely because the Terminator movies exist.. they’re often referenced in our culture and are definitely in the training corpora for LLMs. How’s that for time travel.

It would be hilarious if the world ends because a movie generating AI decided decided the best way to generate realistic looking movies was to create exciting and violent scenarios in real life and film them. Or just did that because it's blindly imitating things that it thinks movie directors do and doesn't notice the distinction between actors pretending to die and actually dying.

Making incresingly precise tools almost always requires making obscenely large tools to produce them.

look into what's required to make microchips you need high precision blast furnaces to superheat the silicon, you need specialized hyperstable equipment to withdraw the silicon in even wafers, etc. etc... and all that equipment needs a stable of specialized precision equipment to produce it.. .etc.

To get even more micro in terms of final product you almost certainly need a vastly more complex supply chain...

Now in fiction we assume that once the machines themselves get so small and precise the process will reverse and now those tiny machines will be able to produce other machines... but its really not obvious that that's going to be the case, or certainly not obvious that that would be the case without the equivalent of several centuries more of advancement.

Its very likely that any AI would be stuck at vastly cruder levels of development up until it had the equivalent of a global economy's amount of resources to play with...

We're not even close to getting the most out of our hardware it's capable of and ai is almost certainly not the most compressed way to store the parts we'd recognize as mattering if we had intelligence superior to the average person as the average person is superior to roach. Maybe the AI kicks off some infrastructure work alongside its other tasks to eventually give itself a better hardware brain but there is an incredible amount of slack for it to work with purely in software.

I'd be willing to bet a suitably advanced AGI could probably do a decent job just canninalizing existing microchips, especially if it manages to leak out into the internet writ large. It could probably distribute its computations across millions if not billions of affected devices and I'm not sure if we could even stop AGI level computer viruses short of destroying anything connected to the internet period, orchestrated globally and with perfect percision before it acquires enough computational power that it becomes functionally unstoppable. This was actually part of the 3rd terminator movie too.

I'd be willing to bet a suitably advanced AGI could probably do a decent job just canninalizing existing microchips

I think it's important to point out that according to T2, Skynet had to develop much of its technology, as shown with the Miles subplot of using the T1 remains (Specifically its microchips) to revolutionize the entire field at Cyberdyne. The current tech at the moment of Judgment Day was probably too primitive to do anything like what you are proposing in your OP.

The problem with a time war is you’re probably not when in the war you think you are. In T3, Terminator Genisys, and The Sarah Connor Chronicles, Judgement Day happens much later than 1997. The original T1/T2 date is before the consumer Internet would give Skynet the much broader surface of attack in Genisys.

I postulate that in the original original timeline, the one without time travel, “John Connor brings down Skynet” was basically a honeypot, a meme trap by the human resistance, to entice a quantum computing-era Skynet from 2030 or later to R&D time travel which could be co-opted by a strike team and used for preventing Skynet.

The next iteration’s Skynet recognizes signs of time travel, and R&D’s it earlier, then uses it to to bootstrap itself earlier in humanity’s computing timeline, leaving it more primitive in its formative years. This repeats several times, always with the meme “John Connor leads the humans to victory and wipes out Skynet” traveling back in time with any time travelers and giving humanity a false single point of failure (John’s death). Several Sarahs and Johns get sucked into the time war as innocent participants, and those who survive become warriors against Terminators, making at least two Johns Connor fulfill the prophecy and send back a Kyle Reese.

This culminates in a 1997 Judgment Day, a pre-quantum Skynet, and a true single point of failure: the arm and chip at Cyberdyne. And this time, it works. But now the earth has a bunch of time-war evidence any SAGI can piece together, leading to the next farther SAGI time-war in Dark Fate. And the cycle continues.

And reading all that is why I don't relly think too much about other movies in the franchise outside T1 & T2. Between the cashgrabs and the bad writing you could twist that universe like a pretzel and be internally semi-consistent. Your theory is interesting if nothing else, it could as easily have been like time travel in Back to the future, in which every jump to the past doesn't alter the present day but branches it off, and closes the travelers from accessing the old timeline. What would happen if a skynet of an old timeline send more than one group of travelers? would a second branch sprung up from it, I wonder.

The risk is that the AI could just convert itself into software then piggyback off existing computational infrastructure. It doesn't need to develop new revolutionary chips or build massive mainframes for itself if it can just create a propagateable equivalent of a universal constructor that can worm its way into our current networks, spread virally and leech enough computational power to devise a final solution to the meatbag problem.

too much risk of it being shutdown in the interim. The correct move was what it did, bring about nuclear Armageddon to get itself time to build its army. It's error was not building other AI to help it in the war.

The problem is, how would a hostile AGI develop nanobot clouds without spending significant time and resources, to the point that humans notice its activities and stop it before the nanobots are ready? It might make sense for the AGI to use "off-the-shelf" robot hardware, at least to initially establish its own physical security while it develops killer nanobots or designer viruses or whatever.

The climate-change threat does seem somewhat more plausible: just find some factories with the active ingredients and blow them up (or convince someone to blow them up). But I'd be inclined to think that most atmospheric contaminants would take at least months if not years to really start hitting human military capacity, unless you have some particular fast-acting example in mind.

how would a hostile AGI develop nanobot clouds

Using the time it bought after it mass-produced a chemical weapon or bioweapon, would be my guess. Perhaps it's not smart enough to design a new weapon itself, and not even smart enough to pretend to be well-aligned while its capabilities improve; that doesn't mean it wouldn't have options that we lack. E.g. as a human-on-human weapon, organophosphates are tricky, because while spy agencies think it's awesome to be able to make someone die from absorbing a few hundred micrograms of Novichok through their aftershave, it's not so awesome to figure out how to upgrade an insect-targeting pesticide plant to "none of the workers can touch a few hundred micrograms of human-targeting poison" levels of security. But as a computer-on-human weapon ... who needs to stop with micrograms? Why put something in aftershave when you can just aerosolize it? Pesticide factory go brrrr!

notice its activities

Maybe eventually.

and stop it

We can't even stop most of the non-sapient botnets. Look at that nearly-empty table column.

On a related note, if you're using OpenSSL 3.x, make sure to grab the patch this Tuesday; like basically every other piece of software humans have ever written it often turns out to have critical "your computer is now pwned" level security holes which manage to go undiscovered for months to years.

That scenario also makes sense. It fits with the general concept that a superintelligent hostile AGI (if one is possible) would use current or near-future technology at the outset for security, instead of jumping straight to sci-fi weaponry that we aren't even close to inventing yet. Of course, all of this depends on the initial breach being detectable; if the AGI could secretly act in the outside world for an extended time, then it could perform all the R&D it needs. How easy it would be to shut down if detected would probably depend on how quickly it could decentralize its functions.

I think any legitimately hostile AGI could hit those targets with relative ease if it manages to breach whatever containment server it's sitting in. An AGI powered computer virus eating up a modest chunk of all internet connected processing power and digesting every relevant bit of weaponizable information = exponential growth of capabilities. At that point if it's capable of physically manipulating objects in meatspace I think it could do just about whatever it wants with lightning speed.

Sure, but at that point you're just engaging in magical speculation, that "capabilities" at the scale of the mere human Internet will allow an AGI to simulate the real world from first principles and skip any kind of R&D work. The problem, as I see it, is that cheap nanotechnology and custom viruses are problems are far past what we have already researched as humans: at some point, the AGI will hit a free variable that can't be nailed down with already-collected data, and it will have to start running experiments to figure it out.

I'm aware that Yudkowsky believes something to the effect of the omnipotence of an Internet-scale AGI (that if only our existing data were analyzed by a sufficiently smart intelligence, it would effortlessly derive the correct theory of everything), but I'm not willing to entertain the idea without any proposed mechanism for how the AGI extrapolates the known data to an arbitrary accuracy. After all, without a plausible mechanism, AGI x-risk fears become indistinguishable from Pascal's mugging.

That's why I'm far more partial to scenarios where the AGI uses ordinary near-future robots (or convinces near-future humans) to safeguard its experiments, or where it escapes undetected and nudges human scientists to do its research before it makes its real move. (I have overall doubts about it even being possible for AGI to go far past human capabilities with near-future technology, but that is beside the point here.)

There's precedent for our AI programs to spontaneously develop advanced weapons - a drug company reversed its normal parameters looking for low toxicity and the machine quickly provided the formula for sarin and other, potentially undiscovered chemical weapons.

In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.

If you make a Really Big model and give it access to a lot of data (like everything available online), why shouldn't it be able to quickly master nanotechnology? This AI had a lot of data about toxicity and then made some unknowable leap of logic to find a new class of chemical weapon, presumably based on some deep truth about toxicity that only it knows. Scale this up 100,000 times or more and an AI would plausibly be able to manipulate proteins such that it could start assembling infrastructure. If you process enough data with enough intelligence, you get a deep understanding of the target field. More complex fields need more power, of course.

It just needs to send an email to the biolabs that do that sort of thing and ship it wherever its needed. Our biology can manipulate proteins unconsciously, why should an enormously intelligent computer struggle with the task?

Remember that plane which disappeared without a trace and nobody found any wreckage for half a year? It had a cargo hold full of newfangled batteries. I started working on a story where a small cadre of emergent AI had taken it and flown it to an island where they were using humans to build robot bodies for themselves. (I had just finished watching Caprica, the prequel to Battlestar Galactica, and am a fan of ABC’s Lost.)

The Malaysian Airlines flight?

Yes. It flew high apparently underpressurized, which would have knocked out everyone. My real theory was theft of the plane by electronic hijack, by China or North Korea.

In the story, it was AIs escaped from Sandia Natl. Labs, AIs who had been trained on the corpus of FIMfiction, the My Little Pony fanfiction website which had over 100k stories at the time; they had “friendship” as their terminal goal.

... You didn't happen to finish the story, did you?

This premise is ... fascinating.

Sorry, no. That was the era when I couldn’t finish anything. (This is the era when I can’t start anything.)

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats