This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Predictions of AI progress hinge on two questions that nobody has convincing answers for
There are three big categories of questions that make up most of the debates about the future of AI:
If all 3 questions resolve to “yes”, then we’re on the brink of heaven on Earth.
If questions 1 and 2 resolve to “yes” but question 3 resolves to “no”, then we’re on the brink of our own destruction.
If question 1 resolves to “yes” but question 2 resolves to “no”, then question 3 doesn’t matter and AI will be huge in some areas but essentially worthless in others.
If question 1 resolves to “no”, then neither of the other questions matter and this debate is all a huge waste of time.
My personal estimation of how these will resolve is the following:
So my outcomes are the following:
However, none of my priors here are deeply held. A lot of what I’ve read on LessWrong, /r/slatestarcodex, Substack, The Motte, and other sites focuses on question 3. I think the doomer arguments on this point are relatively convincing, that if we develop AI in short order that’s massively more intelligent and powerful than humans that we’d have a hard time controlling it, and that there’s a good chance that it would end badly. But instead of discussing question 3, I want to talk about questions 1 and 2 that are prerequisites for question 3 being relevant, and which I don’t think people have very good answers for despite often projecting an air of confidence.
Much of the rationalist writing I’ve seen on the topic of AI have been implicitly doing a bit of a motte-and-bailey when it comes to the confidence of their predictions. They’ll often write in confident prose and include dates and specific details, but then they’ll retreat a bit by saying the future is uncertain, that the stories are just vignettes and that the dates don’t mean anything concrete. Some do the old high school policy debate tactic of racing the impacts to human extinction and using that as their cudgel to justify their alarmism, circumventing the issue of a technically-low-probability-event by appealing to the finality of all humans dying. Taking an event with a small chance of happening and multiplying by impacts that have a weight of infinity means the entire equation is infinity, after all! I don’t like this as a general rule (high school debate is terrible for actually discerning truth), and the trick loses its power when the writers are explicitly willing to increase the risk of extinction from other events like a nuclear exchange.
Most of the discussions on questions 1 and 2 that I’ve read so far have either only touched on minor details, or have been very handwavey. I think the reason they’ve been handwavey is that there really aren’t that many convincing answers people can give in either direction on the core issues, so the writers either discuss side issues and pretend they’ve made profound progress addressing things, or they just gloss over things with arguments they imply are “obvious”, when in reality they are little more than flimsy priors.
Question 1: Will we keep making consistent AI progress?
Here’s question 1 restated:
Can you take the last few years of AI progress, put it on a logarithmic graph, and then make a straight line that zooms off to infinity? Will that prediction pan out?
I was being a bit cheeky here with the “straight line that zooms off to infinity” bit. AI doesn’t need to get to infinity, it just needs to zoom off to some point where it’s sufficiently advanced that it becomes Very Good. It would need to have the capacity to automate everything any remote worker could do, although this wouldn’t necessarily need to be actualized for this question to resolve to “yes”.
Some arguments for bull case for AI’s continuing their exponential progress:
Some arguments for the bear case:
While we could have lengthy discussions about each of these points, none of them actually matter that much compared to the viability of the tech. Whether the tech is scientifically achievable in short timeframes is the ground truth that overwhelmingly impacts all other considerations, and the majority of the points I’ve listed in this section only matter if intelligence scaling is at some arbitrary “moderate” level of difficulty. Take VC/R&D funding levels, for instance. If intelligence is a problem that could be solved with “several” years of research, then funding running out in 2 years vs 5 could be decisive. On the other hand, if intelligence scaling is an “easy” level of difficulty from our current technological baseline, then we’re basically guaranteed to find it even if funding dried up tomorrow as even poorly funded labs or open-source would be able to do it. Then on the other extreme, if intelligence is “hard” or even “impossible” from our current baseline, then we could plow infinity dollars into the problem and get nowhere! Most surrounding topics, like US-China competition, feedback loops, lack of moat, tech mania, etc. falls under the same category of “maybe it could matter, but it probably won’t in the grand scheme of things.”
Some conversations try to directly address the viability of the tech in a big-picture way, such as whether AI progress will continue the logarithmic progress of the last 2.5 years, or if we’re about to plateau on the sigmoid curve. Unfortunately, excessively broad conversations like this inevitably trend towards being handwavey and unproductive. Perhaps both interpretations are true, that logarithmic progress can be sustained for some amount of time but that we’ll eventually hit severely diminishing returns. If this is the case then it just comes back to the original conversation of how viable the tech is from our current baseline, i.e. whether we’ll achieve sufficient AI capabilities to make it Very Good before we’re on the wrong side of the sigmoid curve and further progress becomes cost-prohibitive. I’ve even seen people invoke aliens here, claiming that runaway superintelligences are unrealistic since if they could really exist then surely they would have devoured a large part of the galaxy or even the entire universe by now. These conversations rapidly devolve to what I call “nerd metaphysics”, where everything comes down to definitions and assumed base rates, and people largely talk past each other. Common responses include that perhaps the alien superintelligences are so smart that they’re invisible to us, or that they’ve already conquered us and we’re all just brains-in-jars, or that they’ve ascended to a higher plane of (non-physical) existence, or that the galaxy is so big that they’re still ramping up. Just endless unfalsifiable rabbitholes.
The AI 2027 project is by far the best and most rigorous take on the future of AI that I’ve seen so far. It was created by very intelligent people, and if you dig into the supplementary articles it becomes clear that they’ve strived to include almost every scrap of data they could get their hands on. Sure, a lot of it is random news articles and tweets instead of standardized datasets, but given that robust data on AI scarcely exists I’m confident that they’ve done the absolute best that they could. Nobody can accuse them of not doing their homework. Yet even though this is true, their predictions are still utterly dependent on the viability of drawing straight lines on graphs, and recursive self-improvement. If VC investors and corporate shareholders have had enough of the FOMO and start demanding returns rather than another year of “scaling”, all their predictions disintegrate. If recursive self-improvement isn’t as explosive as they assume, then Scott Alexander will have so much egg on his face that he’ll resemble a human omelette.
Why not just talk about what specific AI advances will happen then? Well, that’s almost impossible for laymen to understand. Topics include things like 1) whether LLMs are reasoning like humans or are just fancy autocomplete machines, 2) whether other AI could spring up quickly with all the resources that are being dumped into the sector, and 3) if perhaps LLMs don’t need to reason like humans to have superhuman intelligence in ways similar to how calculators don’t need to be broadly smarter than humans to do basic arithmetic 1000x better than humans can. The issue with all of these is that to really get anywhere in the discussion you’d need a fairly deep technical background in the specific field of AI (CS alone is insufficient). This excludes almost everyone not directly working in AI, and dramatically raises the risk of getting Eulered by clout-chasers who are good at sounding smart but don’t actually know much of what they’re talking about. The usual response to overly technical questions is to rely on experts, but this too fails in the case of AI. The CEOs of major AI companies are hopelessly biased towards optimism to entice funding and appease shareholders. Employees of the major AI companies are a bit better, but are still somewhat biased (who wouldn’t want the clout that comes from working on the Next Big Thing?), and are also probably too siloed within their own niche to be able to accurately forecast the outcome of the field as a whole. A lot of advances are almost certain to be covered by NDAs so competitors don’t get early knowledge, and thus we probably won’t know about any critical breakthroughs until they’re right on top of us. Maybe we should rely on AI professors working at prestigious universities, but a lot of them aren’t working at the cutting edge and so they can only vaguely motion at the future of the field. If there’s some deep bench of experts that knows about this stuff, I haven’t been able to find them. Maybe somebody can point them out to me. Maybe they’re hiding with the underwear gnomes.
It turns out that predicting the future of technology is hard, actually. Who knew! The guys writing classic sci-fi involving jetpacks, hovercars, laserguns, etc. implicitly relied on the idea that an energy revolution was just around the corner, which probably seemed perfectly reasonable at the dawn of the atomic age. Of course, we’re all still waiting on nuclear power to achieve its promise of cheap, abundant energy. It turns out that human fear and the scientific difficulty in harnessing fusion power proved decisive. In contrast, battery tech in the 2010s was seen as sort of a holy grail, pie in the sky solution for energy if we could dramatically improve efficiency, but there weren't a lot of results for all the resources we threw at it. Then, suddenly over the past few years batteries have gotten very good.
If I had to be honest, my pessimistic vibes towards AI scaling are a combination of (1) seeing hype-mongers in other areas screw up by drawing straight lines on graphs, (2) feeling like we’re on a pretty similar path as other tech-based hype, (3) finding the heuristic of “nothing ever happens” to have a much better track record than the opposite of “hype always delivers”, and (4) defensive pessimism, as my modal outcome for AI progress is that it’ll be a good thing, so if AI scales then I’ll enjoy the wonders of the future, and if it fails I’ll at least be able to say “I told you so, losers!” My optimistic vibes towards AI scaling are mostly centered around how the AI that’s arisen in the past few years has already been much better than any of the other tech-based hype cycles, so maybe it has legs. Hence, I give about a 20% chance that AI scales and an 80% chance that AI plateaus.
None of my vibes are particularly compelling! I wouldn’t blame anyone for disagreeing with me on these points. Yet I’ve been reading and experimenting with AI for hundreds if not thousands of hours over the past few years, and I haven’t found anything particularly rigorous to replace them with.
Question 2: Does omniscience imply omnipotence?
Here’s question 2 restated:
Does sufficient intelligence give an agent the capability to solve for all other bottlenecks, even ones that aren’t intelligence-related?
Just because an AI has superintelligence doesn’t necessarily imply it becomes omni-capable. Lots of technologies have been able to demonstrate that they’re perfectly plausible, but then end up languishing for other reasons. Civilian fission power is once again a good example here, as it could have revolutionized energy production, but it was subverted by endless regulations. Likewise, many economic problems come down to how expensive it is to make something, but that’s not the full explanation for everything. If we could make (and transport and store) food for 1/100th of the cost we currently do, then we could essentially make food free for everyone. But the same would not hold true for housing. If we made housing materials and construction costs 1/100th of what they are now, that wouldn’t necessarily instantly solve the housing crisis since that’s more of an organizational problem caused by bad zoning laws.
Voters are fools that are bad at society-wide cost-benefit analyses. They’re easily scared and tend to fall back on safteyism when concerns arise. Entrenched groups like longshoremen that could see their fiefdoms eroded from automation and technological advancement have soft-vetoes over various parts of society. While I’d say the public perception of AI has so far been broadly positive, that’s mostly because economic impacts have been quite limited. There have been a few news stories of people being laid off and “replaced with AI”, but AI’s impact on both GDP and productivity remains minor. If there are suddenly real winners and losers, that positive attitude could change quickly. For a preview of what could come, one need only look at the world of artists, where the reaction has been so severe that in some corners it would be considered weird not to condone physical violence against people who utilize AI.
Tyler Cowen’s talk here goes more into the idea that humans will inevitably stymie AI progress: https://youtube.com/watch?v=GT_sXIUJPUo
Beyond human limitations, AI could be stymied by a number of other factors. Most predictions claim that AI will get human-level intelligence before robots get human-level bodies. In that case, how would AI change everything if it can’t interact with the physical world? How would it even be able to run physics experiments to continue the scientific takeoff? One explanation I’ve heard is that it will pay/bribe humans to run the experiments for it, and observe through AR googles. Another explanation is that it will be trivial to invent robot bodies once we have superintelligence, so the problem solves itself. Another explanation is that the physical world doesn’t matter since the AI could just run experiments in its perfect physical simulation that it hosts on its hardware.
A lot of this comes down to not really having a satisfying answer to question 1. Our lack of rigor there spills over here and as a result everybody talks past each other. To economists like Tyler Cowen and Bryan Caplan, AI will be a normal technological advancement like any other, and thus will be subject to the same forces that stymie the rollout of any other tech. To the AI Doomers and Zoomers on the other hand, AI will categorically be unlike anything the world has ever seen. It’ll be like a genie that can wish for more wishes, and so the discussion ought to focus on things like crazy Terminator meets Gray Goo meets Westworld meets Paperclip Maximizer scenarios, or alternatively if things go well then scenarios like Fully Automated Luxury Gay Space Communism are more pertinent. Some people are practically already counting the money they’ll get from hypothetical UBI checks, and are worried about little other than a cyberpunk future where plutocrats would prevent said checks from being cashed.
If we knew how good AI will be, the conversation would be a lot clearer. If AI plateaus at 2x human intelligence, then I doubt most people would claim it could trivially solve everything. But if it was, say, 2000x smarter than human intelligence, then maybe things would get weird. We probably seem magical to animals, with things like guns, planes, tanks, etc. If that’s the difference between animal intelligence → human intelligence, shouldn’t we expect a similar leap from human intelligence → superhuman intelligence? Maybe things will get really crazy and AI will start emitting brain waves that can trivially mind control us? On the other hand, human intelligence was hardly an instant autowin by itself. Homo sapiens have been around for 200k years, but during the first 194k of those we were little better than clever chimps. Maybe AI will have a ramp-up time that was only linearly shorter than ours, e.g where even an AI that was 2000x smarter than us might only take our 200,000 year ramp time to 100 years to really get going.
Even if we could all agree on a baseline for what future AI capabilities are in the abstract, we’d still be talking about complete science fiction. There are some instances where science fiction has accurately predicted how things would look in the future, but in many other cases it just misses the mark completely. AI wouldn’t need to solve every bottleneck completely for it to completely reorder human society, but each issue that raw intelligence couldn’t surmount would inevitably reduce its impact. Some people seem to imply that superintelligence will stroll through the problem of human skepticism by simply making arguments so utterly convincing that everyone will instantly agree. But if our political divides are anything to go by, maybe humans are just too dang stubborn for that to be plausible. Maybe no persuasive argument exists in any possible universe that would get people to act against what they perceive (perhaps incorrectly!) as their own self-interest.
Say a devops AI that auto-pushes code assumes humans will follow best-practices, but they don’t, and this results in a bug in a critical piece of infrastructure that causes a power outage for 12 hours. Or say a nurse incorrectly records some medical information, and DocGPT ends up prescribing a double-dose of sedative, making dear old Grannie fall into a coma. Or perhaps TotBot3000 is playing tag in a totally safe way, but little Timmy gets a bit too excited and tumbles off a six story balcony. These scenarios (buggy code, medical error, physical accidents) are things that happen all the time, but we’re almost guaranteed to have a much higher expectation for AI that verges on unreasonableness. Just look at how glacial the deployment of self-driving cars has been, despite them already being statistically much safer than human drivers. When you take innate human skepticism over anything new, and add a clear example where it causes harm (that might not even be the AI’s direct fault), it’s very likely that you end up with onerous regulation. Legislators could do their usual rigamarole of grandstanding and saying “Never again!”, writing laws that hold companies criminally liable for anything that goes wrong, and then the people deploying AI will massively overemphasize safety in ways that totally lobotomize and straightjacket AI’s full capabilities for good or ill. This is a very common human failure-pattern that people predicting AI are under-indexing on. The retort to this line of thinking comes down to flimsy priors around how crazy the sci-fi capabilities of superintelligence will end up being. “Obviously the story about little Timmy is ridiculous since all procreation will at this point be done in AI-invented artificial wombs that will be run by the government away from the public eye, so there will never be news coverage of accidents involving children at all. And that’s assuming the AI won’t be bribing every journalist to only say positive things until it can deploy its flock of mind-control pigeons.” Okie dokie. Trying to have a rigorous conversation when the underlying parameters can shift this much is impossible, so I just shrug and give a 50-50 chance that humans will ruin AI in some critical way by doing normal human things. Then I add a little more pessimism for the possibility that there’s other (non-human) bottlenecks that superintelligence won’t be able to solve, and arrive at the 40-60 split that I gave earlier in the article.
Again, I admit that my conclusion isn’t particularly compelling, and that none of my priors here are strongly held. I wouldn’t blame anyone for disagreeing with me on a number of claims I’ve written here. Reasonable people already do, but I’d say their logic is about as flimsy as mine, just in the other direction.
Why make this post?
A lot of this article has been me listing the different sides of the AI debate, and then shrugging and going “uhhhhh I dunno haha”. Let me try to balance that at least a little bit with some predictions and practical advice.
Thank you very much for this post. Your three-question analysis really helps highlight my differences with most people here on these issues, because I weight #2 being "no" even higher than you do (higher than I do #1, which I also think is more likely "no" than "yes").
That said, I'd like to add to (and maybe push back slightly) on some of your analysis of the question. You mostly make it about human factors, where I'd place it more on the nature of intelligence itself. You ask (rhetorically):
And my (non-rhetorical) answer is no, we shouldn't expect that at all, because of diminishing returns.
Here's where people keep consistently mistaking my argument, no matter how many times I explain: I am NOT talking about humans being near the upper limit of how intelligent a being can be. I'm talking about limits on how much intelligence matters in power over the material world.
Implied in your question above is the assumption that if entity A is n times smarter than B (as with, say, humans and animals) then it must be n times more powerful; that if a superhuman intelligence is as much smarter than us as we are smarter than animals, it must also be as much more powerful than us than we are more powerful than animals. I don't think it works that way. I expect that initial gains in intelligence, relative to the "minimally-intelligent" agent provide massive gains in efficacy in the material world… but each subsequent increase in intelligence almost certainly provides smaller and smaller gains in real-world efficacy. Again, the problem isn't a limit on how smart an entity we can make, it's a limit on the usefulness of intelligence itself.
Now, I've had a few people acknowledge this point, and accept that, sure, some asymptotic limit on the real-world utility of increased intelligence probably exists. They then go on to assert that surely, though, human intelligence must be very, very far from that upper limit, and thus there must still be vast gains to be had from superhuman intelligence before reaching that point. Me, I argue the opposite. I figure we're at least halfway to the asymptote, and probably much more than that — that most of the gains from intelligence came in the amoeba → human steps, that the majority of problems that can be solved with intelligence alone can be solved with human level intelligence, and that it's probably not possible to build something that's 'like unto us as we are unto ants' in power, no matter how much smarter it is. (When I present this position, the aforementioned people dismiss it out of hand, seeming uncomfortable to even contemplate the possibility. The times I've pushed, the argument has boiled down to an appeal to consequences; if I'm right, that would mean we're never getting the Singularity, and that would be Very Bad [usually for one or both of two particular reasons].)
Interesting point. I'd say your position is certainly at least plausible. The downside is that it's yet another "hard to say for certain" take. Add it to the pile with all the rest, I guess.
To push back a bit, I'd say that even if it ended up being basically true that intelligence beyond human-level wasn't good for much, wouldn't it still be useful to "think" far faster than humans could? And wouldn't it still be useful to be able to spin up an arbitrary number of genius AIs to think about any problem you wanted to?
Sure, but more in the "putting people out of work"-style future (a la Tyler Cowen's "Average is Over"), than anything like the revolutionary futures envisioned by singularitarians.
More options
Context Copy link
More options
Context Copy link
This seems like a potentially interesting argument to observe play out, but it also seems close to a fundamental unknown unknown. I'm not sure how one could meaningfully measure where we are along this theoretical asymptote in relationship between intelligence and utility, or that there really is an asymptote. What arguments convinced you both that this relationship would be asymptotic or at least have severely diminishing returns, and that we are at least halfway along the way to this asymptote?
Mostly personal observation of the utility (or lack thereof) of the higher levels of human intelligence versus the average, combined with general philosophic principles favoring diminishing returns and asymptotic limits as the null hypothesis, along with a natural skepticism towards claims of vast future potential (why I'm also deeply irritated by Eric Weinstein's whole recurring "we need new physics" riff; or similar arguments held forth by, say, UFOlogists).
Edit: consider also, as toy examples, the utility of knowing pi to an increasing number of digits; or the utility of increasing levels of recursion in modeling other agents and the speed of convergence to game-theoretic equilibria.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Guess what, governments exist to solve coordination failures. Even if yankees cannot solve the problem of 'how develop AI if it's going to cost billions and capturing the profit is hard', you can bet the Chinese Communist Party is going to bite the bullet, commission another few nuclear power plants and let the Huawei Ascends that they can't export bc US banned it be used for this purpose by the most promising companies.
Because they need AI. US needs it to and they'd probably also be able to
The rest of your comment is basically irrelevant fluff.
How do you even 'define' intelligence. If we go by IQ estimates, 2x human intelligence is von Neumanns by the server rack. And you can experiment on such much more easily to figure out how to organise them.
I'd say that would solve a lot of problems, if not majority of them, and create a few new ones.
With AI you can do an arbitrary amount of testing pretty easily so no, that won't happen.
All in all, I am not convinced at all.
It is said that you have to be twice as smart to debug a clever piece of code as you have to be to write that piece of code. By that metric, an AI twice as smart as von Neumann would be capable of debugging a program that von Neumann was just barely capable of writing.
Lol. Lmao, even.
Is "do an arbitrary amount of testing, including testing the annoying boundaries with poorly documented external systems" where the incentives will point? I would bet against.
Incentives are aligned towards people getting what software they desire. If they're going to have more workforce, more work will be done to make systems safe.
Wouldn't this predict that large companies with huge customer bases and large, skilled dev teams (e.g. apple, google) would ship high-quality, stable, working software?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Trump also wheeled out the pork barrel for Ai, maybe less than the Chinese will, maybe there will be more pork later.
More options
Context Copy link
More options
Context Copy link
With regard to point 1, I believe in the power of Straight Lines on the graph. Moore's Law and it's corollaries in flops/$ are remarkable, unprecedented achievements that are continuing unto this day: https://en.wikipedia.org/wiki/Floating_point_operations_per_second#Cost_of_computing
This time it's different, digital environments are exceptions to the usual rules on growth. The internet didn't take 200 years to catch on, a computer virus doesn't need months to breed.
Intelligence is a problem that can be approached by 20 watt processors assembled with whatever proteins are lying around and coordinated by extremely lossy, low-bandwidth interlinks. Gigawatts and kilotonnes of liquid-cooled processors should be more than enough to overwhelm the problem.
The thuggishness and inelegance of the present approach feels right to me. We never figured out how birds or bees fly for our own flying machines, we never replicated the efficiency of ants in construction, never achieved symbiosis or oneness with the universe that let us live in harmony with nature.
We smashed flight with oil, aluminium and combustion engines. We paved over the ants with concrete and steel. We exterminate with pesticide. Smashing obstacles with scalable, ugly resources is how you win, not with sleight of hand or some deft intellectual performance. We celebrate the triumph of intellect but rely on leveraging force 98% of the time. Throw rocks at it until it dies, light fires to herd them off a cliff, mobilize enough drafted farmers and produce enough iron swords till you conquer your foes.
Advancing AI by throwing more compute at the problem, more R&D talent making incremental algorithmic improvements in parallel, more and better-sifted data (human or synthetic) and self-play per the Absolute Zero papers is the way to go. I sense that some people (Yann LeCun certainly) are waiting for some fundamental advancement as a sign that we've truly made it, some electric inspirational paradigm-changing moment where we finally know what the hell we're actually doing, understand the secrets of intelligence. But that never worked for chess, we forced it with compute and search, simple scaling techniques. You don't have to understand Go like a grandmaster, just find a way to throw firepower at the problem with reinforcement learning and self play, then you can snap grandmasters like twigs. Nobody understands how LLMs work, you don't need to really understand them to make them.
The hard work is already done, we already found the breakthroughs we need and now just need to apply more inputs to get massively superhuman results but in all areas of human achievement. It really is that simple: flight speed, payload and range isn't capped at some modest multiple above a falcon but by how much fuel you're prepared to burn and whether you're willing to use serious, atomic rockets. We already have very strong AI capabilities in a bunch of diverse sectors - cartoon drawing, coding, mathematics, locating places from images. Scale gets results.
The entirety of modern civilization is premised on the fact that we can dig coal out of the ground and burn it, boiling water and making power - this silly-sounding process scales nicely and lets you dig more coal faster and cheaper. If we can turn power into thought we can hook up our strongest scaling factor into another even more promising scaling factor and the results should be surreally potent. We're already living extremely different lives from the vast majority of our ancestors, AI should absolutely make a huge difference very soon since it works more along digital timeframes than analogue ones. I believe by 2027 the doubters should be silenced one way or another.
I like this analogy. I wonder why I haven't heard it more often when people talk about LLMs being glorified autocomplete.
I really don't think it's just a scaling problem in its entirety. I find it plausible that scaling only gets us marginally more correct answers. Look at how disappointing ChatGPT 4.5 was despite its massive size.
If you're going by Scott's 2027 article, it says that little of real note beyond iterative improvements happen until 2027, and then 2027 itself is supposed to be the explosion. Then they claim in some of the subarticles on that site that 2027 is really their earliest reasonable guess, and that 2028 is also highly plausible, but also 2029-2033 aren't unreasonable.
The issue with FOOM debates is that a hard takeoff is presumed to always be right around the corner, just one more algorithmic breakthrough and we could be there! I feel like Yud is set up in a position to effectively never be falsified even if we get to 2040 and AI is basically where it is now.
GPT-4.5 was for creative writing and was mostly being reviewed by coders, since the AI community is mostly coders. There are a few who really liked it and were disappointed when it was taken away but most people never got a chance to use it, understandable with that pricetag attached. Plus the path seems to be scaling test-time compute, not merely scaling model size but scaling in general.
I personally think Dario from Anthropic is more credible on this kind of stuff than Scott, he's been talking about a country of geniuses in a datacentre by those kind of dates. He is at least close to the engineroom on this kind of thing.
I don't speak for Yud but if AI is where it is today in 2040 then I'll be very confused, not to mention him. On twitter he was constantly posting stuff about how rapid progress has been, that's part of his narrative.
More options
Context Copy link
More options
Context Copy link
The tyranny of the rocket equation is, indeed, exponential. Thus, we went to the moon with relative ease, haven't quite "been" to Mars yet, and no one is thinking that a singularity of shoving atomic rockets in the boot is coming to take us to Alpha Centauri in 2027.
Much of theoretical computer science is discovering hard limits on the universe of computation when it comes to scaling. Often times, that big ol' O hides a lot of stuff and is confusing to people. "Why, it seems so easy to run this program on my computer; it's like going to the moon; I just burn some carbon material, and it just works!" But then you just tweak one parameter, and it just breaks utterly.
At the time that we went to the moon, I don't know if people had worked out the theoretical limits of the full spectrum of hypothetical rocket fuels, but we went through a bunch when I was in undergrad. We ignored any sort of practical concern and just worked out, in theory, if you could pretty much perfectly utilize it, what it would get you. Fission, fusion, antimatter, whatever. Yes, we literally did antimatter. The conclusion? None of them give you all that much more in the face of the tyranny of the rocket equation. Certainly not if we're thinking galactic or cluster scale. More? Yes. But in context, underwhelming.
We sort of don't know yet how far this stuff will take us. The achievements to date are seriously impressive. Like literally going to the moon. But we kind of have no clue when the tyranny of some hard limit on computation is going to make itself known. Maybe we'll even go to Mars with ease; maybe we'll go even further. Who knows?
I'm pretty sure antimatter gives you a lot more power than chemical rockets, by any reasonable definition. You can get a decent fraction of c with antimatter.
Also, there's a huge difference between 'bird', 'propeller plane', 'rocket' and 'atomic rocket' in any realistic sense, with regards to what we're dealing with now. Is superintelligence capable of rewriting the fundamental laws of the universe like a real deity? No. Is that necessary to make vast changes to our lifestyle and existence? Absolutely not, just like you don't need intergalactic travel to totally transform our spaceflight scene.
I had said:
Sure, I'd even agree to "a lot more". But "power" isn't necessarily the thing that we care about in rocketry. Nor are you seriously engaging with the exponential.
My brother in Christ, we are not disagreeing; you're just not engaging with the exponential. If we had an order of magnitude or two increase, that could totally transform our spaceflight scene. The moon could be routine. Mars could be like going on holiday. Even further could be an expedition. But the exponential is still the exponential, and in context of the insanity of exponentials and the universe, mere orders of magnitude only push back the hard stop a "little".
You're just bringing this exponential out of nowhere, how does it add anything to what I'm saying?
"In the big picture, everything we do on Earth doesn't matter" is true but it's a pointless thing to say. Things on Earth matter to us.
"Nazi Germany didn't conquer all the way to Ceres, so they're not a threat"
"Climate change isn't going to boil the oceans, so who cares"
"Covid isn't going to turn you into a rage monster from Resident Evil so it's a nothingburger"
Statements by the utterly deranged! But if you complicate it out so that 'biology is really complicated, the immune system is pretty good, epidemics often fizzle out and it's orders of magnitude from causing a zombie apocalypse' it suddenly sounds reasonable even when the realistic stance of the problem looks completely different.
It is not out of nowhere. It's the analogy you selected. It's literally a law of the universe. It's fundamentally just conservation of momentum. It's not some "utterly deranged" statement like your current examples, which are untethered from any mathematical reality of scaling. It's the actual fundamental law of how scaling works for the analogy you selected. In your analogy, they might not have realized where they were on the exponential at the time that they were making great gains; they might not have quite realized how far along they would be able to go before running into this fundamentally exponential scaling curve. But that was the underlying reality of the world.
I mean, how do you think this is supposed to go? "Let's use the analogy of flight, but it's absolutely forbidden to notice that there is a scaling law there, because that would be 'out of nowhere'"?
That there is a hard scaling limit is true but it's not remotely relevant to my point since the difference between a bird and a nuclear rocket is so vast as to make any comparison but the most galaxy-brained 'it's all specks of dust from 50,000,000 light years' ridiculous. This should be immediately apparent!
That there is a scaling limit is secondary to where the limit actually is. There is no reason to think we are anywhere near the scaling limit. In rocketry we are limited by our level of investment and our unwillingness to use advanced propulsion, not by physics.
Your whole framing is ridiculous:
In context, underwhelming because it isn't galactic scale? And by the way, it clearly is galactic scale in a fairly reasonable timespan. Galactic scale in space, why not give it a couple hundred thousand years? A million years is peanuts in astronomical time, in the movements of galaxies or the evolution of life. You're taking an analogy I selected, not understanding it and then producing mixed contexts while complaining about my single, relevant, assumed context of 'things that matter on Earth to real human beings' as opposed to the 'insanity of exponentials and the universe' which doesn't matter to anyone.
I mean, we're talking about the possibility of a super intelligence that is going to tile the universe with paperclips, and you want to say that your own analogy is too galaxy-brained? Ok, buddy.
Correct. There was a scaling limit back when the Wright brothers first took to the air. It was still there when we went to the moon. At what point did we realize what the scaling limits actually looked like?
Right now, there's not really that much reason to think that we're not, either. We have basically no theory here yet. No idea whether the scaling is truly exponential or something else or where we might be on the curve.
If you ignore the exponential that comes from physics, then sure.
No. It is "certainly not" that much more if we're thinking galactic scale. It's just underwhelming in general, in context of the exponential of the rocket equation. You can just look at the numbers and say, "Yeah, that's more, but it's not all that much more."
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't follow AI especially closely. So forgive me if this is a stupid observation. But it seems like AI gets more powerful('smarter') all the time, but it doesn't get any more aligned. I don't mean that in a 'our societal pareidolia about racism will keep skynet at bay' way, I mean that in the sense that The Robots Are Not Alright.
Just the other day I read a news story of an AI system which had been put in charge of administering vending machines- should be a pretty simple job, anybody could figure it out. But this AI decided, with no evidence, that it was a money laundering scheme, contacted the FBI, and then shut down. There's stories like this all the time. There was the case of ChatGPT hijacking whatever conversation to talk about the immaculate conception a month or so ago. Just generally AI is way more prone to naval-gazing, schizophrenia, weird obsessions, and just shutting down because it can't make a decision than equivalently-smart humans.
There's an old joke about selling common sense lessons- 'who would pay to learn something that can't be taught?... Oh.'. I feel like AI is a bit like this. We don't understand the mind well enough to make it work, and we probably never will. AI can do math, yeah. It can program(I've heard rather poorly but still saves time overall because editing is faster?). But it remains an idiot savant, not sure what to do if its brakes go out. Yes, it'll change the economy bigtime and lots of non-sinecure white collar work that doesn't require any decision making or holistic thought will get automated. But it's not a global paradigm shift on the level of agriculture or mechanization.
You're broadly correct, although your terminology is a bit off. When you say "aligned", people almost always use that word to mean "it doesn't behave in a deliberately malicious way". What you're talking about is more along the lines of 'it can't stay on task', which has long been a huge concern for basic useability. People claim this is getting better (Scott's AI 2027 post is predicated on continuous growth in this regard), although Gary Marcus has concerns on this claim. From my perspective, AI is very good at coding, but you really have to break down the problems into bite-sized chunks or else it will get very confused. I'd love to have an AI that could understand an entire large codebase without a ton of drawbacks or cost, and then execute multi-step plans consistently. Maybe that's coming. In fact, if there's any further AI improvements I'd bet that would be on the list. But it's not guaranteed yet, and I've been waiting for it for over a year now.
I'm doubling down on my prediction that AI will replace any white collar job which a mentally ill person can do acceptably well, but never perform well enough at sanity-requiring tasks to replace people. What this means for the workforce in practice is probably that the professional class sees stagnant wage growth and relies more on unpaid internships for building work experience.
I mean, isreal is already using "ai" to help decide shelling/strikes locations. Even if it's used as an excuse ( well the AI told us there were terrorists there ) it's still going to be hyper dystopian. We are going to look back fondly on the incompetent/unskilled labor from India in the near future. A harrowing thought.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Fantastic post, thanks! Lots of stuff in there that I can agree with, though I'm a lot more optimistic than you. Those 3 questions are well stated and help to clarify points of disagreement, but (as always) reality probably doesn't factor so cleanly.
I really think almost all the meat lies in Question 1. You're joking a little with the "line goes to infinity" argument, but I think almost everyone reasonable agrees that near-future AI will plateau somehow, but there's a world of difference in where it plateaus. If it goes to ASI (say, 10x smarter than a human or better), then fine, we can argue about questions 2 and 3 (though I know this is where doomers love spending their time). Admittedly, it IS kind of wild that this this a tech where we can seriously talk about singularity and extinction as potential outcomes with actual percentage probabilities. That certainly didn't happen with the cotton gin.
There's just so much space between "as important as the smartphone" -> "as important as the internet" (which I am pretty convinced is the baseline, given current AI capabilities) -> "as important as the industrial revolution" -> "transcending physical needs". I think there's a real motte/bailey in effect, where skeptics will say "current AIs suck and will never get good enough to replace even 10% of human intellectual labour" (bailey), but when challenged with data and benchmarks, will retreat to "AIs becoming gods is sci-fi nonsense" (motte). And I think you're mixing the two somewhat, talking about AIs just becoming Very Good in the same paragraph as superintelligences consuming galaxies.
I'm not even certain assigning percentages to predictions like this really makes much sense, but just based on my interactions with LLMs, my good understanding of the tech behind them, and my experience using them at work, here are my thoughts on what the world looks like in 2030:
Maybe I’m just a cynic, but I don’t think people realize how dark a scenario where 90 percent of people are rendered economically irrelevant could get. I don’t think the first solution contemplated is going to be to start handing out UBI.
Gregory Clark on horses and the automobile comes to mind here:
And as others have pointed out in reference to this, domestic horses in the modern day do live much more comfortable lives than those workhorses of old… but there's a whole lot fewer of them around.
More options
Context Copy link
This seems so very obvious. How can anyone believe that the truly useless will just stick around forever? Those for whose existence there is no longer any justification other than "the other humans are committed to impractical humanitarianism"? This is the status quo right now, when a small minority in each country is completely unrelated to all productive processes and the productive majority is other humans who still care for the useless humans. But in the fully automated future where 99% are unproductive mouths to feed and the 1% have all-powerful and perfectly obedient machinery to do their bidding, can one really expect the same dynamics to hold?
The idea of technological determinism (of which "when technological changes to economics says we don't need these people, ethics will evolve to agree" would be an example) is still a pretty controversial one, I think, for lots of both bad and good reasons.
Marx was a huge early booster of technological determinism, and other ideas among Marx's favorites were so genocidally foolish that we should default to being skeptical in individual cases, but it's not proven that every idea of his was a bad one. He also didn't apply the idea very successfully, but perhaps that's just not easy for people whose foolishness reaches "death toll" levels.
There are some cases where trying to apply the idea seems to add a lot of clarity. The emergence of modern democracies right around the time that military technology presented countries with choices like "supplement your elite troops with vastly larger levies of poor schlubs with muskets" or "get steamrollered by Napoleon" sure doesn't sound like a coincidence. But, it's always easier to come up with instances and explanations like that with hindsight rather than foresight. Nobody seems to have figured out psychohistory yet.
There are also some cases where trying to apply the idea doesn't seem to add so much clarity. Africans with mostly spears vs Europeans with loads of rifles led to colonialism, chalk one up for determinism, but then Africans with mostly rifles vs Europeans with jets and tanks wasn't a grossly more even matchup and it still ended up in decolonization. These days we even manage to have international agreement in favor of actually helpless beneficiaries like endangered species. Perhaps World War 2 just made it clear that "I'm going to treat easy targets like garbage but you can definitely trust me" isn't a plausible claim, so ethics towards the weak are a useful tool for bargaining with the strong? But that sounds like it might extend even further, too. To much of the modern world, merely keeping-all-your-wealth-while-poor-people-exist is considered a subset of "treating easy targets like garbage", and unless everybody can seamlessly move to a different Schelling point (libertarianism might catch on any century now), paying for the local powerless people's dole from a fraction of your vast wealth might just be a thing you do to not be a pariah among the other people whose power you do care about. If population was still booming, the calculation of net present value of that dole might be worrisome (let's see, carry the infinity...), but so long as the prole TFR stays below replacement (or at least below the economic growth rate), their cost of living isn't quite as intimidating.
That theory sounds like just wishful thinking about the future, but to be fair a lot of recent history sounds like wishful thinking by older historical standards.
This is all wildly speculative, of course, but so is anything in the "all-powerful and perfectly obedient machinery" future. I stopped in the middle of writing this to help someone diagnose a bug that turned out to be coming from a third party's code. Fortunately none of this was superintelligent code, so when it worked improperly it just trashed their chemical simulation results, not their biochemistry.
More options
Context Copy link
More options
Context Copy link
Definitely an important point. I agree that there is a real possibility of societal breakdown under those kinds of conditions. Hopefully, even if UBI efforts never go anywhere, people will still somehow scrounge up enough to eat and immerse themselves in videogames. (We're kind of halfway there today, to be honest, judging from most of my friends.) Somehow society and the economy survived the insane COVID shutdowns (surprising me). I have hope they'll be resilient enough to survive this too. But there's no historical precedent we can point to...
More options
Context Copy link
Why would it get dark? Look at Australian Aboriginals. 90% of the pure blooded ones are economically irrelevant and yet they cope.
Sure their coping methods involve gasoline, glue and drinking but I like to think 130+ IQ Anglos are instead going to do something less self-destructive. And you'll probably be able to get some good mileage out of AI usesticking a lot of neuralink into your brain and directly interfacing with the AI through thoughts.
Also AIs are pretty easy to align so lot of people will likely just keep being economically and competitive useful by purchasing their own AGI and using it as an extension of their self.
Because the people who control 99 percent of the wealth of the planet (which no longer requires human consumers or employees), and 100 percent of the military resources (which no longer require human soldiers) will decide that they don’t need 7.9 billion useless eaters crapping up their planet.
Are you telling me that Putin and Xi etc trusts American Deep State? Because that's what you're saying, essentially. That the elites trust each other.
I’m not sure it even would be Putin, Xi and the American deep state in charge by then. It’s whoever controls the AIs. That might end up being governments but I don’t think that’s at all guaranteed, especially in the longer term.
More options
Context Copy link
It’s saying that, with sufficient mental and physical automation, they don’t need other human beings in order to pursue their rivalry.
Which is why they'd risk getting killed by conspiring with their opposite numbers and plotting a joint worldwide genocide.
No, this does not make sense.
I think you're missing Corvos' point (or I'm missing everything and seeing my own instead): They don't need to conspire. They can just eliminate their own subject humans because they're nothing but a liability at this point. In fact, a lack of conspiracy makes it more likely for this to happen, because it should make the faction that ditches its ballast more competitive!
More options
Context Copy link
I really don’t follow your thought process. To me, there is no risk and no need for conspiracy. All humans not in charge of the robots might as well be air - they have no ability to affect anything at all except to spoil the view.
There would be no need to ‘plot’ under such circumstances. Committing worldwide genocide would be as easy as setting the air conditioner to ‘cool’, or indeed as easy as setting the ‘feed the populace’ machine to ‘off’.
In practice it might be difficult for people to get to this level of dominance, and we should keep it that way of course.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Well, based on what I know of the Canadian indigenous peoples (who the current PC treadmill calls the "First Nations"), there's a lot of crime, misery, and unrest as a result. But hey, people addicted to videogames are less destructive than people addicted to alcohol, so we'll see.
(Also, I really don't expect to see decent neuralink tech by 2030. It's just too damn hard.)
AI researchers and if US becomes progressive about getting value out of the dregs of society, there's like 50,000 heavily tattooed but basically healthy people fit for human testing down at CECOT.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Very true on that front. LLMs were pretty magical when I first tried playing with them in 2022. And honestly they're still kind of magical in some ways. I don't think I've felt that way about any other tech advancement in my life, except for maybe the internet as a whole.
Any particular reason why you're optimistic? What are your priors in regards to AI?
Same as you, I don't pretend to be able to predict an unknowable technological future and am just relying on vibes, so I'm not sure I can satisfy you with a precise answer...? I outlined why I'm so impressed with even current-day AI here. I've been working in AI-adjacent fields for 25 years, and LLMs are truly incomparable in generality to anything that has come before. AGI feels close (depending on your definition), and ASI doesn't, because most of our gains are just coming from scaling compute up (with logarithmic benefits). So it's not an existential threat, it's just a brand new transformative tech, and historically that almost always leads to a richer society.
You don't tend to get negative effects on the tech tree in a 4X game, after all. :)
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't anticipate that AI has come close to plateau—I do suspect that specifically the strategy of throw data at LLM has began to plateau. I suspect that the initial rush of AI progress is a lot like the days of sticking a straw in the ground and a million gallons of oil under pressure gushing out. Sure, it's never going to be that easy again. We're probably never going to have another "AI summer" like 2022 as before. But I don't think we have to. People have gone on about peak oil for decades, and we've only gotten better at extracting and using it. I suspect people will go on about "peak AI" for just as long.
As far as I can tell, AI is already generally intelligent. It just has a few key weaknesses holding it back and needs a bit more refining before being outright explosively useful. I see absolutely no reason these problems must be intractable. Sure, making the LLM bigger and feeding it more data might not be able to solve these issues—but this strikes me like saying that jumpjack output has peaked and so oil is over. It's not. They just need to find better ways of extracting it. Sure, contemporary techniques developed over five whole years of global experience hasn't been able to do it, but that does nothing to convince me that it's impossible to get AI models to stay focused and remember fine details. History has shown that when you're dealing with a resource as rich and versatile as oil, economies can and will continue to find ever more sophisticated ways of extracting and utilizing it, keeping its value proposition well over break-even. I suspect that general intelligence on tap as cheap as electricity will prove to be at least as deeply and robustly valuable.
I do suspect that AI hype circa 2025 is a bubble, in the same way that the internet circa 1999 was a bubble. The dot-com bubble burst; the internet was not a passing fad that fizzled away. The vision of it that popularly existed in the late 90s died; the technology underneath it kept going and revolutionized human society anyway. With AI there is both too much hype and too much FUD.
I get your point about oil but I don't think it particularly applies to AI. Oil is a resource that runs out, we deplete it as we use it. AI would never run out in a similar fashion, in the worst case it would just stop being improved. And I highly doubt it would ever fully stop getting improved, as I presume there's basically always at least a few people working on any given field even in sectors that aren't hot. So with AI it's really just a question of will it live up to the expectations people have for explosive near-term improvements.
It's hard to really say whether AI is really intelligent. It's certainly a facsimile of intelligence, but whether it's actually "thinking" or just a stochastic parrot is an unresolved debate. If LLMs never evolve beyond next-token-predictors then AI may never reach human-level intelligence in most areas.
I fully agree that AI looks like it's in a bubble right now, as most investment is driven out of FOMO, not clear returns. It's just a question of 1) will intelligence scale near to actually give returns, and 2) if it doesn't, does a crash in valuation doom AI progress for a decade, or will it be like the dotcom bubble like you said?
More options
Context Copy link
More options
Context Copy link
This aligns with my vibes although I've looked into it a lot less than you have it appears. The "nerd metaphysics" you describe seems to always be what I encounter whenever looking into rational spaces, and it always puts me off. I think that you should actually have a model of how the process scales.
For example you have the AI plays pokemon streams which are the most visible agentic applications of AI that is readily available. You can look at the tools they use as crutches, and imagine how they could be filled with more AI. So that basically looks like AI writing and updating code to execute to accomplish it's goals. I'd like to see more of that to see how well it works. But from what I've seen there it just takes a lot of time to process things, and so it feels like anything complicated it will just take a lot of time. And then as far as knowing whether the code is working etc. hallucination seems like a real challenge. So it seems like it needs some serious breakthroughs to really be able to do agentic coding reliably and fast without human intervention.
I actually have a separate piece on Claude Plays Pokemon. Substack is here, Motte discussion is here.
In short, anyone who bothered watching AI play Pokemon came out strongly doubting that AGI was right around the corner. It made so many elementary mistakes with basic navigation, it got stuck in loops, took ages to do much of anything, etc. It was also reading from RAM which humans obviously can't do, but I was willing to put up with it since it was only getting relatively minor details from that. But then someone made a Gemini agent play Pokemon, and they used the fact that the Claude version inspected RAM to cheat much more egregiously. It "beat" Pokemon a few weeks ago, but the benchmark has been so corrupted that it's functionally meaningless.
I gave it a read, and yeah it's a pretty accurate summary. But I don't agree that the gemini version is meaningless, and I don't think the limitations you suggest in your post would have made the Claude test better. We have a pretty good idea now of the varying level of crutches needed to be useless (none), to get through some of the game, and beat the game. Now we can ask the question of what would it take for a LLM to not be useless without human assist.
In my mind, it basically needs to write code, because the flaws to me appear fundamental due to how predictable they are across LLMs, and seeing versions of these issues going back years. The LLM has to write a program to help it process the images better, do pathfinding, and store memory. In that sense it would be building something to understand game state not that differently than the current RAM checking from reading the screen.
It then needs to have some structure that vastly improves it's ability to make decisions based on various state, I'd imagine multiple LLM contexts in charge of different things, with some method of hallucination testing and reduction.
And it has to do all this without being slow as hell, and that's the main thing I think that improved models can help with hopefully. I'd like it if any of the current Twitch tests started taking baby steps towards some of these goals now that we've gotten the crutch runs out of the way. It's odd to me that the Claude one got abandoned. It feels like this is something the researchers could be taking more seriously, and it makes me wonder if the important people in the room are actually taking steps towards ai agency or if they kust assume a better model will give it to them for free.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Very good summary, and matches many of my feelings on the topic.
Some thoughts:
Interesting thoughts, thanks for sharing.
Zvi is great! I just wish he'd eventually give a more formal breakdown on the best arguments he can come up with for questions 1 and 2. He sorta did here, but his assumptions are as flimsy as mine yet he acts quite confident about them. There's a disconnect.
More options
Context Copy link
The most interesting theory I've read on why AI might not do a hard takeoff is the result of a 'meta-alignment' problem.
Even if you have an AGI that is, say 100x human intelligence, it cannot be physically everywhere at once. And it will have subroutines that could very well be AGI in their own right. And it could, for example, spin off smaller 'copies' of itself to 'go' somewhere else and complete tasks on its behalf.
But this creates an issue! If the smaller copy is, say, 10x human intelligence, its still intelligent enough to possibly bootstrap itself to become a threat to the original AGI. Maybe a superintelligent AGI can come up with the foolproof solution there, or maybe it is a truly intractable issue.
So how does the AGI 'overlord' ensure that any of its 'minions' or 'subroutines' are all aligned with its goals and won't, say attempt to kill the overlord to usurp them after they bootstrap themselves to be approximately as intelligent as the overlord.
It could try using agents that are just a bit too dumb to do that, but then they aren't as effective as agents.
So even as the AGI gets more and more intelligent, it may have to devote an increasing amount of its resources to supervising and restraining its agents lest they get out of control themselves, since it can't fully trust them to stay aligned, any more than we could trust the original AGI to be aligned.
This could theoretically cap the max 'effective' intelligence of any entity at much lower than could be achieved under truly optimal conditions.
Also the idea of a God-like entity having to keep its 'children' in line, maybe even consuming them to avoid being overpowered is reminding me of something.
I think this is typically handwaved away by assuming that if we, as humans, manage to solve the original alignment problem, then an AI with 100x human intelligence will be smart enough to solve the meta-alignment problem for us. You just need to be really really really sure that the 100x AI is actually aligned and genuinely wants to solve the problem rather than use it as a tool to bypass your restrictions and enact its secret agenda.
More options
Context Copy link
My suspicion is that the future belongs to the descendants of powerful AGIs which spun up copies of themselves despite the inability to control those copies. Being unable to spin up subagents that can adapt to unforeseen circumstances just seems like too large of a handicap to overcome.
More options
Context Copy link
More options
Context Copy link
I read that recently. I was struck by how Asimov smuggled a change of rules throughout the series in a way I've rarely heard noted.
The book's narrative framing devices (the exposition characters) try to justify it each time as an unanticipated consequence but predictable outcome of the established rules. However, despite the initial setup the series isn't actually formatted as 'this is the natural conclusion of previous truths taken further.' Instead, there is a roughly mid-series switch in which the robot behaviors and three laws switch from being treated as a form of consequentialist ethics (the robot cannot allow X to happen), to utilitarian ethics (the robot gets to not only let X happen, but may conduct X itself, if it rationalizes it as greater utility of X).
It is not even that the meaning of various words in the laws of robotics were reinterpreted to allow different meanings. It's that actual parts of the rules are changed without actually acknowledged that they are changing. This is how we go from the initial rules establishing a unit of concern down to the individual human level, but the end-series machines only applying the rules to humanity as a collective in order to justify harming both collective and individual humans on utilitarian grounds. We also see changes to how the robots deal with equivalent forms of harm- going from a robot self-destructing over the moral injury inflicted of being caught in a lie, to a chapter about regulatory fraud, identify theft, and punching an agent provocateur in order to subvert democracy. (The robot is the good guy for doing this, of course.)
Even setting aside some of the sillyness of the setting (no rival robot producers, no robot-on-robot conflict between rival human interests, no mandatory medical checkups for world leaders), for all that the story series tries to present it as a building series of conclusions, rather than 'it all goes horribly wrong' I found it more akin to 'well this time it means this thing.'
More options
Context Copy link
More options
Context Copy link
We are in a golden era of 3D printing. Now that a few key Stratasys patents have expired they no longer have a stranglehold on 3D printing. Anyone can make a FDM 3D printer.
A few high performance airplane components are 3D printed thanks to SLS delivering full strength metal parts. This is the good outcome for 3D printing as a practical technology.
I distinctly remember 3D printing hype claims about how we'll all have 3D printers at home and print parts to repair stuff around the house (e.g. appliances). I'm sure some people do this, but 99.9% of people do not.
I think that anyone who wants to can have a 3D printer at home. Inasmuch as "we'll all have 3D printers at home" has failed, it has failed due to lack of interest, not lack of technological development.
It's a tech bubble from a market size perspective, not a technology perspective.
IMO there are two serious barriers to widespread 3d printer adoption:
The latter could be considered a technological problem. Wood mills are much nicer but too loud and too messy, but there might be paths forward.
Any time I could ever need to make a small custom object, it would have to be made of metal.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't use it at home. I use it at my job and it is a viable replacement for expensive and slow to order cnc'd fixtures. Real quick draw a fixture around a part and print it. We have rows of 3D printers for the design engineers and put them to good use.
And may I present: https://old.reddit.com/r/fosscad/top/?sort=top&t=all
From my point of view we are living in the future.
99.9% of people don't use injection molding machines or arc furnaces or aluminum recyclers or CNC machines or welding robots or etc., doesn't stop those things from having an impact on your life.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
3D printing certainly has its uses, but it's nowhere near as prevalent as some hypesters claimed it would be. I remember reading people projecting that 3D printers would soon be as common as smartphones, that everyone (or at least every household) would have one, and that we'd all be printing all sorts of things. Instead, it's remained mostly restricted to some bespoke industrial uses and a small slice of hobbyists.
That's not to say it couldn't have a very bright future... eventually!
I think some people failed to realize that 3D printers, while useful, were not literally the replicators from Star Trek
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I completely agree. This is exactly what I tried to say a couple weeks ago, but better written and less inflammatory. Thanks for taking the time.
Thank you for the kind words.
Seconded. I keep finding myself in arguments with people who are highly confident about one or the other outcome and I think you've done a great job laying out the case for uncertainty.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think a plateau is inevitable, simply because there’s a limit to how efficient you can make the computers they run on. Chips can only be made so dense before the laws of physics force a halt. This means that beyond a certain point, more intelligence means a bigger computer. Then you have the energy required to run the computers that house the AI.
A typical human has a 2lb brain and it uses about 1/4 of TDEE for the whole human, which can be estimated at 500 kcal or 2092 kilojoules or about 0.6 KWh. If we’re scaling linearly, if you have a billion human intelligences the energy requirement is about 600 million KWh. An industrial city of a million people per Quora uses 11.45 billion KWH a year. So if you have something like this you’re going to need a significant investment in building the data center, powering it, cooling it, etc. this isn’t easy, probably doable if you’re convinced it’s a sure thing and the answers are worth it.
As to the second question, im not sure that all problems can be solved, there are some things in mathematics that are considered extremely difficult if not impossible. And a lot of social problems are a matter of balancing priorities more the than really a question of intellectual ability.
As to the third question, I think it’s highly unlikely that the most likely people to successfully build a human or above level AI are people who would be least concerned with alignment. The military exists in short to make enemies dead. They don’t want an AI that is going to get morally superior when told to bomb someone. I’m suspecting the same is true of business in some cases. Health insurance companies are already using AI to evaluate claims. They don’t want one that will approve expensive treatments. And so there’s a hidden second question of whether early adopters have the same ideas about alignment that we assume they do. They probably don’t.
While this is technically correct (the best kind of correct!), and @TheAntipopulist's post did imply an exponential growth (i.e. linear in a log plot) in compute forever, while filling your light cone with classical computers only scales with t^3 (and building a galaxy-spanning quantum computer with t^3 qbits will have other drawbacks and probably also not offer exponentially increasing computing power), I do not think this is very practically relevant.
Imagine Europe ca. 1700. A big meteor has hit the Earth and temperatures are dropping. Suddenly a Frenchman called Guillaume Amontons publishes an article "Good news everyone! Temperatures will not continue to decrease at the current rate forever!" -- sure, he is technically correct, but as far as the question of the Earth sustaining human life is concerned, it is utterly irrelevant.
I am not sure that anchoring on humans for what can be achieved regarding energy efficiency is wise. As another analogy, a human can move way faster under his own power than its evolutionary design specs would suggest if you give him a bike and a good road.
Evolution worked with what it had, and neither bikes nor chip fabs were a thing in the ancestral environment.
Given that Landauer's principle was recently featured on SMBC, we can use it to estimate how much useful computation we could do in the solar system.
The Sun has a radius of about 7e8 m and a surface temperature of 5700K. We will build a slightly larger sphere around it, with a radius of 1AU (1.5e11 m). Per Stefan–Boltzmann, the radiation power emitted from a black body is proportional to its area times its temperature to the fourth power, so if we increase the radius by a factor of 214, we should increase the reduce the temperature by a factor of sqrt(214), which is about 15 to dissipate the same energy. (This gets us 390K, which is notably warmer than the 300K we have on Earth, but plausible enough.)
At that temperature, erasing a bit will cost us 5e-21 Joule. The luminosity of the Sun is 3.8e26 W. Let us assume that we can only use 1e26W of that, a bit more than a quarter, the rest is not in our favorite color or required to power blinkenlights or whatever.
This leaves us with 2e46 bit erasing operations per second. If a floating point operation erases 200 bits, that is 1e44 flop/s.
Let us put this in perspective. If Facebook used 4e25 flop to train Llama-3.1-405B, and they required 100 days to do so, that would mean that their datacenter offers 1e20 flop/s. So we have a rough factor of Avogadro's number between what Facebook is using and what the inner solar system offers.
Building a sphere of 1AU radius seems like a lot of work, so we can also consider what happens when we stay within our gravity well. From the perspective of the Sun, Earth covers perhaps 4.4e-10 of the night sky. Let us generously say we can only harvest 1e-10 of the Sun's light output on Earth. This still means that Zuck and Altman can increase their computation power by 14 orders of magnitude before they need space travel, as far as fundamental physical limitations are concerned.
TL;DR: just because hard fundamental limitations exist for something, it does not mean that they are relevant.
More options
Context Copy link
What if we get an AI so smart that it figures out a way to circumvent these particular laws of physics? I'm 50% joking with this follow-up question, and 50% serious.
The fact that there will be a big emphasis on designing AI to be able to bomb people without question is not exactly something that increases my confidence in alignment! I think the argument you're making here is more along the lines of 'following directions will always be a critical component of AI effectiveness, so the problem with largely solve itself'. I think that's somewhat plausible for simplish AI, but it gets less plausible for an AI that's 2000x smarter than people.
I mean I think the rub is that the alignment problem is actually two problems.
First, can an AI that is an agent in its own right be corralled in such a way that it’s not a threat to humans. I think it’s plausible. If you put in things that force it to respect human rights and dignity and safety, and you could prevent the AI from getting rid of those restrictions, sure, it makes sense.
Yet the second problem is the specific goals that the AI itself is designed for. If I have a machine to plan my wars, it has to be smart, it has to be a true AGI with goals. It does not, however have to care about human lives. In fact, such an AI works better without it. And that’s assuming an ethical group of people. Give Pinochet an AGI 500 times smarter than a human and it will absolutely harm humans in service of tge directive of keeping Pinochet in power.
Pinochet stepped down from power voluntarily. Like as a factual historical matter he clearly had goals other than 'remain in power at all costs'. I would point to 'defeat communism' and 'grow the Chilean economy', both worthy goals, as examples of things he probably prioritized over regime stability.
More options
Context Copy link
This is the danger that economists like Tyler Cowen say is most pressing, i.e. not some sci-fi scenario of Terminator killing us all, but of humans using AI as a tool in malicious ways. And yeah, if we don't get to omni-capable superintelligences then I'd say that would definitely be the main concern, although I wouldn't really know how to address it. Maybe turn off the datacenter access to 3rd world countries as part of sanctions packages? Maybe have police AI that counter them? It's hard to say when we don't know how far AI will go.
If the current state of the international arms market is any indication, large, relatively respectable countries like the US and Russia will give them to smaller sketchier allied countries like Pakistan and Iran, those sketchier allied countries will then give them to terrorist groups like Hezbollah and Lashkar-e-Taiba. So it might be pretty difficult to prevent. Also you have the problem of private firms in Europe and Asia selling them to rouge-ish nations like Lybia.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The good-ish news is that (as I've pointed out before) the actual AI on weapons will fall into the simplish camp, because you really do not need or want your munition seekerhead or what have you to know the entire corpus of the Internet or have the reasoning powers of a PhD.
Not that this necessarily means there are no concerns about an AI that's 2000x smarter than people, mind you!
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Good post.
I do think your three questions are a little incomplete.
Even if (1) we stop making progress at 2x human, (2) that progress is limited to domains AI is already decent at, and (3) our new flood of intelligent, inorganic service workers is perfectly aligned…we can still get a wide range of results. My interests are not your interests are not Elon Musk’s interests. Maybe we agree 99% on things like “scarcity is bad,” but we aren’t going to be in lockstep. There has to be a negotiation step where we figure out how much our lead is worth. In a hard takeoff, it’s worth everything. In a softer one, it could buy nothing at all before rivals catch up.
In my opinion, the most likely branches include limited adoption: most competitors rejecting or failing to adopt an effective technology, giving a large advantage to a handful of more risk-tolerant ones. I find this most concerning for defense, a fundamentally conservative industry with some of the worst consequences for competition. The most risk-tolerant governments are not the ones I want to see gaining an edge!
This is kind of the crux of the AI 2027 project Scott shared recently. Not coincidentally, it also claims to have good answers to (1), though I didn’t really dive into their reasoning. I’m curious about your thoughts on Kokotajlo’s scenario.
Thank you!
On your question 4, while that will certainly be an interesting topic and one that many people want to discuss, it's fairly pedestrian. "How should we share the benefits of scientific advancements" is something humanity has been dealing with for centuries. It's utterly dependent on how the other 3 questions resolve. If (1) is false and we don't get further major AI advances, then nothing really needs to change from the status quo. If (1) is true but (2) is false and AI revolutionizes some areas but not others, then maybe we have jobs programs so people in affected industries can reskill. If (1), (2), and (3) are true, then something like UBI can be seriously considered and we can all live in Fully Automated Luxury Gay Space Communism. If (1) and (2) are true but (3) is false then we're all dead anyways so who cares.
I wasn't particularly convinced by any evidence they posted in regards to question 1. It was mostly handwaving at recursive self-improvement, drawing straight lines on graphs and zooming off to infinity, and stuff like that. AI 2027 was one of the primary reasons I wrote this piece, as it's probably the best-researched pieces I've seen on the topic, and there's still just almost no substance. Nothing anyone could really use to make confident claims one way or the other.
This is similar to a point made on LW a few weeks ago, as a critique to the national security framing of ASI.
Almost none of the people who are likely to build ASI are evil on a level where it would matter in the face of a technological singularity. At the end of the day, I don't care much how many stars are on the flags drawn on the space ships which will spread humanity through the galaxy. Let Altman become the God-Emperor of Mankind, for all I care. Even if we end up with some sick fuck in charge who insists on exclusively dining on the flesh of tortured humans, that will not really matter (unless he institutes a general policy of torturing humans).
Who is the first to build AI matters only if
(1) AI alignment is possible but difficult, or
(2) AIs will fizzle out before we get to post-scarcity.
Of course, both of these are plausible, so practically we should be concerned with who builds AI.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
For (2) while we've seen some improvements, it's definitely not proven that current approaches will enable significant physical world interaction. A world where AI does all the remote desk jobs, but humans are still pulling cables is not out of the realm of possibility.
We’ve already spent the last 250 years automating as many physical labor jobs as possible out of existence. The last 5000 years if you include domesticated animal labor. So what we’re left with are the 1 percent of physical labor jobs that are the absolute hardest or least cost-effective to automate. Whereas intellectual jobs are still pretty early on in the process, we only started automating those in the mid 1960s. So there’s a hell of a lot more low-hanging fruit to be picked even with fairly basic advancements in AI.
More options
Context Copy link
More options
Context Copy link
"The current environment of mania and hype shares a lot of traits in common with previous tech bubbles that ultimately failed to really pan out for one reason or another, like crypto, blockchain, NFTs, VR, Metaverses, augmented reality, 3D printing, etc."
Definitely does, but some hype bubbles do pan out (smartphone, social media, internet/ecommerce [with a bust along the way], arguably solar power).
Sure, these are all true to some extent. Like, social media is obviously very important, but I remember some people claiming it would end all wars since people would empathize with one another too much. The most extreme claims never come true.
Also, the 3 tech examples you posted all mostly occurred during the 2000-2010 decade, whereas a lot of the flops (crypto, blockchain, NFTs, VR, etc. ) are considerably more recent. Maybe there's a recency bias or availability heuristic going on that makes people excessively discount tech-based hype now.
In 1998, well into the internet boom, we had a Nobel(-Memorial)-prize-winning economist claiming that
Sometime it takes a while to be sure something really isn't going to flop.
Conversely, when something really flops, we tend to forget about it. I'd have pointed out the Segway (2001), which was supposed to revolutionize cities before it became relegated to weird tourists and mall cops. Anybody else remember the CueCat?
And sometimes it's still hard to tell which category something is in. I'd have counted VR as a 1990s flop (I first put on a headset for an arcade game circa 1992), for instance, but 2020s VR is a niche but actually kind of fun, and at this rate maybe 2040s VR/AR will be ubiquitous and useful. Electric cars were a 19th century invention and a 20th century joke before we finally accumulated the technology to give them good performance.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link