site banner
PaperclipPerfector  1yr ago (text post)   32999 thread views

Culture War Roundup for the week of April 17, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

8
Top Bottom New Controversial Comments
Jump in the discussion.

No email address required.

Is the rapid advancement in Machine Learning good or bad for society?

For the purposes of this comment, I will try to define good as "improving the quality of life for many people without decreasing the quality of life for another similarly sized group" an vice versa.

I enjoy trying to answer this question because the political discourse around it is too new to have widely accepted answers disseminated by the two American political parties being used to signify affiliation like many questions. However, any discussion of whether something is good or bad for society belongs in a Culture War threat because, even here on The Motte, most people will try to reduce every discussion to one along clear conservative/liberal lines because most people here are salty conservatives who were kicked out of reddit by liberals one way or another.

Now on to the question: Maybe the best way to discover if Machine learning is good or bad for society is to say what makes it essentially different from previous computing? The key difference in Machine Learning is that it changes computing from a process where you tell the computer what to do with data, and turns it into a process where you just tell the computer what you want it to be able to do. before machine learning, you would tell the computer specifically how to scan an image and decide if it is a picture of a dog. Whether the computer was good at identifying pictures of dogs relied on how good your instructions were. With machine learning, you give the computer millions of pictures of dogs and tell it to figure out how to determine if there's a dog in a picture.

So what can be essentialized from that difference? Well before Machine Learning, the owners of the biggest computers still had to be clever enough to use them to manipulate data properly, but with Machine Learning, the owners of the biggest computers can now simply specify a goal and get what they want. It seems therefore that Machine Learning will work as a tool for those with more capital to find ways to gain more capital. It will allow people with the money to create companies that can enhance the ability to make decisions purely based on profit potential, and remove the human element even more from the equation.

How about a few examples:

Recently a machine learning model was approved by the FDA to be used to identify cavities on X-rays. Eventually your dental insurance company will require a machine learning model to read your X-rays and report that you need a procedure in order for them to cover treatment from your dentist. The justification will be that the Machine Learning model is more accurate. It probably will be more accurate. Dentists will require subscriptions to a Machine Learning model to accept insurance, and perhaps dental treatment will become more expensive, but maybe not. It's hard to say for sure if this will be a bad or a good thing.

Machine learning models are getting very good at writing human text. This is currently reducing the value of human writers at a quick pace. Presumably with more advanced models, it will replace commercial human writing all together. Every current limitation of the leading natural language models will be removed in time, and they will become objectively superior to human writers. This also might be a good thing, or a bad thing. It's hard to say.

I think it's actually very hard to predict if Machine Learning will be good or bad for society. Certain industries might be disrupted, but the long term effects are hard to predict.

I believe there are two broad scenarios for what might happen with ML from an economics/politics perspective.

Scenario 1 is that ML will be a powerful productive tool (ie capital) in the hands of those that can afford it just like many other inventions throughout history.

If this happens the reaction will be along the lines we all know too well. The left will complain that those in power gain even more power and now have novel ways to control and/or extract value from workers. Plus a lot more low-skilled people will become unemployable and redundant so class tensions will probably get worse. On the flip side a few smart early movers will make insane bank and shape the way the next few decades will go. Could be interesting to see how different nation states adopt the new technology.

Scenario 2 is the "things get crazy" scenario. What if ML takes off far quicker than people are expecting, for example by recursively improving itself? I believe in that case we might be unable to fit the development into our usual political lens. If one company has twice as much capital than everyone else combined our systems of power distribution fall apart. If one nation has capabilities that make it effectively invincible our models for foreign relations stop working. If that happens it will be more akin to a scenario where superintelligent aliens have landed on earth and all bets are off.

What if ML takes off far quicker than people are expecting, for example by recursively improving itself? I believe in that case we might be unable to fit the development into our usual political lens. If one company has twice as much capital than everyone else combined our systems of power distribution fall apart. If one nation has capabilities that make it effectively invincible our models for foreign relations stop working.

I'm a little disconcerted at how many people who are working in the industry seem to hold this as the explicit goal and are intentionally maneuvering things so as to prevent anyone from intervening until it's too late.

I expect that "recursively improving itself" will lead to the AI going off into the weeds -- that is, evolving in ways unconnected to the real world. The output will quickly become bizarre and not particularly useful. It works for formal systems like Go because the rules are well-defined, but you can't simulate reality to a sufficient degree of precision.

I think the idea behind recursive self-improvement is more like, a 150 IQ AI should be able to find a way to increase its IQ to 151, a 151 IQ AI should be able to increase its IQ to 152, and so on and so forth until it reaches godhood.

It doesn't necessarily have to simulate large portions of reality, if it's able to find a way to isolate the factors responsible for its g factor and come up with a generalized way of making improvements to those factors. Presumably as part of the cycles of improvement it could interact with the real world in order to get more training and data. But this sort of scenario has its own issues.

Especially if it can spin up various copies of itself and make minute changes to see how that effects performance. Basically massive, parallel experimentation.

I am strongly of the opinion that since neoliberal PMC jobs are the easiest to automatic with AI, there will be incredibly strong regulation banning AI from taking the jobs of the PMC. The power to regulate is the power to destroy, and as incapable of actual productivity the PMC and their legion of bullshit jobs are, they know how to run a grift and bask in their own self importance.

No, what you need to fear from AI is when Facebook fires up an instance of AutoGPT for each user and tasks it with keeping them doom scrolling for as long as is possible. If you thought "the algorithm" was already amoral and sanity shredding, you ain't seen nothing yet. That was a mere baby, feebly hand tuned by meat that thinks (or thinks it thinks). When the AI is fully unleashed on slaving our attention spans to our screens, it's going to be like how Fentanyl turbo charged opioid deaths. You're gonna start seeing people literally starving to death staring at their phones. Actually, nix that, they'll die of dehydration first. I momentarily forgot that nearly always happens first.

I'm gonna register this prediction now too. Apparently Ai has trouble with fingers. You'll know it's gotten loose when there is a new tiktok trend of young people amputating all their fingers. The AI will have decided it's easier to convince us to get rid of our own fingers than figure out how to draw them better. Given the rates of Tiktok induced mental illness, it would probably be right in that assessment.

I am strongly of the opinion that since neoliberal PMC jobs are the easiest to automatic with AI, there will be incredibly strong regulation banning AI from taking the jobs of the PMC. The power to regulate is the power to destroy, and as incapable of actual productivity the PMC and their legion of bullshit jobs are, they know how to run a grift and bask in their own self importance.

This is exactly why the crossbow and handgonnes never took off and why we still live under a feudal system ruled over by our lieges and ladies.

More seriously, this technology is too valuable to not use, anyone who does use it is going to gain a massive advantage over anyone that doesn't, its use is inevitable.

More seriously, this technology is too valuable to not use, anyone who does use it is going to gain a massive advantage over anyone that doesn't, its use is inevitable.

The same is true of nuclear power. It's the only technology that will allow us to hit emission targets and keep the grid stable with cheap, reliable power.

But we've built 3 nuclear power plants in as many decades, and our infrastructure is crumbling and less reliable than ever. Our ruling class simply does not care so long as they can keep living that 0.01% life. Even now they are setting preposterous 10 year EV targets, despite not putting a dime towards building out a domestic EV supply chain or infrastructure. Including upgrading our electric grid to deal with the massive increase in demand all those EVs will create. Which brings us back to the nuclear power they scorn so much.

Your appeals to a reasonable nation performing certain obvious reasonable tasks are pointless. This is clown world. You need to think dumber.

Nuclear power has a lot of benefits, but it takes a significant amount of time and money to get online, with the benefits being generally diffused. The number of organisations that can actually get a nuclear power plant online for long enough that they can start to make a profit is quite small.

AI is comparatively cheap, the changes are quick and easily observable and the pay off for an individual willing to utilise it is substantial. As a class medievial European nobility may have benefited from a complete ban on crossbows and handguns, but the ratio of costs to return of employing these weapons meant that anyone who chose to defect and take up their use would out compete those who did not. The same is true of AI, it cannot be ignored.

Your appeals to a reasonable nation performing certain obvious reasonable tasks are pointless. This is clown world. You need to think dumber.

I'm appealing to human greed and desire for power. You need to think smarter.

The same is true of nuclear power. It's the only technology that will allow us to hit emission targets and keep the grid stable with cheap, reliable power.

Exactly. The general population believes what it was told for 50 years - nuclear power is something immensely dangerous and deadly, something that can explode at any moment, kill millions and turn the whole country into uninhabitable desert full of motocycle riding mutants.

Now, imagine if normies are told:

THE COMPUTER can kill you. Yes, THE COMPUTER can shred you into paperclips, without warning. And not only you, but everyone, everyone in the whole world. Yes, even ordinary computer in your son's room can do it.

Do not wait for your doom. Say something, do something.

The general population believes what it was told for 50 years - nuclear power is something immensely dangerous and deadly, something that can explode at any moment, kill millions and turn the whole country into uninhabitable desert full of motocycle riding mutants.

Again, at least according to this poll, 76% of Americans - the most relevant demographic for this forum - favor nuclear energy. Even the opponents do not necessarily hold the most alarmist and charged view of nuclear as a power source.

The American public won't give up guns, do you think they'll give up computers?

Heck, even if it's just AIs they're told to give up, forces that want to do that will have to move fast, because every passing moment it reaches more hands, and the hands that have it are gonna hold on tight. And at some point soon, we will reach a point of cultural no return on everyone having these tools.

The problem is, we've already had hacker scares for years, I don't know what it would really take for people to realize the threat outside of re-hashed Terminator references.

I'm gonna register this prediction now too. Apparently Ai has trouble with fingers. You'll know it's gotten loose when there is a new tiktok trend of young people amputating all their fingers. The AI will have decided it's easier to convince us to get rid of our own fingers than figure out how to draw them better. Given the rates of Tiktok induced mental illness, it would probably be right in that assessment.

This would be a rad short story. An AI that gets 'frustrated' at its own limitations against the real world and it's solution is to just sand off all the sharp edges that are giving it problems.

Like it genetically engineers all the cows to be spherical so it's physics simulations can be more accurate.

An AI that gets 'frustrated' at its own limitations against the real world and it's solution is to just sand off all the sharp edges that are giving it problems.

I'm obligated to point out that this already happened, the AI was capitalism, the sharp edges were all direct human interactions, and our atomized broken society is the result.

I would be interested in seeing this thought/analogy expanded.

Seeing Like a State plus a broad view of what constitutes a "state," perhaps?

I thought I had seen later Scottposts applying this logic to capitalism.

His Meditations on Moloch sounds like this vein too.

I thought I got this idea from Mark Fisher or Nick Land, but random googling isn't leading me to any obvious writing of theirs on this specific concept. Come to think of it maybe it was one of IlForte's pithier comments. Regardless you should read both of them.

I am strongly of the opinion that since neoliberal PMC jobs are the easiest to automatic with AI, there will be incredibly strong regulation banning AI from taking the jobs of the PMC. The power to regulate is the power to destroy, and as incapable of actual productivity the PMC and their legion of bullshit jobs are, they know how to run a grift and bask in their own self importance.

I highly doubt this will happen. You talk as if the PMC is a giant union where everyone is aligned, which shows you don't understand the social context there and are clearly just poo-pooing your outgroup.

People in the PMC with power have capital, whether it's political, intellectual, or financial. The financial movers and shakers will not agree to regulating AI, at least until they have gotten their piece of the pie. Even if they do, it will take years and years to get everyone to agree on a framework.

You've also got the AI companies themselves. Altman has come out and said he doesn't think regulation at this stage is a good idea, and he's got an incredible amount of political and intellectual capital. Many people in government, for good reason, see Altman as one of the most important figure in the world right now. They don't want to piss him off.

The advent of generative AI heralds the single largest change in the structure of human society since the neolithic revolution (ie. the invention of agriculture and the settled society) 12,000 years ago.

I would actually argue this is a closer parallel to the cognitive revolution, or homo sapiens first discovery of culture, language, and general cognitive technology. The difference is that with the revolution from fire, or the agricultural revolution, or the industrial revolution, or even the internet, the AI revolution deals with intelligence and a paradigm of thinking itself. The Scientific revolution could also be a close contender, since it dramatically increase our ability to think and use our knowledge.

The only thing that seems certain is that it will radically reshape the life of every single human being on earth in the next 5-50 years.

Agree strongly here.

Currently, we're focused on the application of modern LLMs and other generative models to create media (writing, images, video etc) and to perform knowledge roles that involve a combination of text and data manipulation and basic social interaction (ie. the vast majority of PMC labor sometimes derogatorily referred to as 'email jobs'). But current models are so generalizable, and LLMs already appear to translate so well to robotics that even relatively complex physical labor is only a few years behind the automation of the PMC, especially given rapid improvements in battery technology and small motors, which are some of the other major bottlenecks for robotic labor.

The real step change in my opinion is once these models get good at things like drug discovery, mathematical proofs, and building models of physics. We have essentially been locked into a paradigm almost 100 years old in physics, and haven't found many fundamental changes in mathematical or chemical theory since then either, to my knowledge.

In the past, every time we had a major breakthrough in one of these fields it was enough to reshape the world entirely. Chemistry led to the industrial revolution, Newtonian Mechanics led to the scientific revolution. (or was the beginning, whatever.)

There is a (relatively persuasive) case to be made that the invention of agriculture led to a decline in the quality of life for the vast majority of human beings that lasted until the late 19th or early 20th century. It took 11,900 years for the neolithic revolution to pay quality of life dividends, in other words. We can only hope that the period of relative decline in quality of life is shorter this time round, or perhaps avoidable altogether.

As I mention above, I think the comparison to the agricultural revolution falls flat for a number of reasons. Admittedly most revolutions follow a pattern of short term negative issues with long term positive outcomes however.

For the purposes of this comment, I will try to define good as "improving the quality of life for many people without decreasing the quality of life for another similarly sized group" an vice versa.

Tangential, but the term in economics you are touching here is a Kaldor–Hicks improvement I think. It's not Pareto-optimal, but total-wealth increasing, and could theoretically be converted to a Pareto-optimal situation with redistribution from the winners to the losers (assuming such redistribution does not have any externalities itself!).

Personally am very excited for AI improvements. I’m hoping something like ChatGPT will be able to act as a super personal assistant and analyst.

For example in personal life, would love to be able to type into a box that I’m looking to plan a trip with just a few parameters (date, general budget, etc) and have it send me options. I can then have the AI send even more options for what do on the trip and finally book reservations that only require my approval.

That’s just one example but there are plenty of admin type activities that I’d like to offload to an AI. The opportunities in professional life are even greater but I think that may take longer as the aversion to giving the AI access to confidential data may be high (it’s currently banned at my mega corp).

I’m hoping something like ChatGPT will be able to act as a super personal assistant and analyst.

At what level of 'smarts,' however, will an AI that is already training on how you do your job going to stop needing you around to do it?

I mean, you're basically happily accepting an apprentice who will lighten your workload whilst learning your job, except this thing is known to learn 100x faster than your standard human. The assumption that you'll have moved on to bigger and better things (or retired) before the apprentice steps up to take over your job may not hold here.

I think at least in the short/medium term this technology could lead to large productivity gains without corresponding cuts in total headcount.

When I started my career finance teams used to forecast in excel using relatively simple formulas. Now they use coding languages and forecast more frequently, with greater detail, and greater accuracy while working with massive data sets. This hasn’t lead to a huge cut in overall headcount, but it has changed the skill set mix on the teams.

Right, but it's presumably cheaper to spin up more GPT instances or build up more datacenters than it is to train more 'experts' in fields that are susceptible to ML automation.

Hence the question:

At what level of 'smarts,' however, will an AI that is already training on how you do your job going to stop needing you around to do it?

I'm not really doubting that humans will be 'in the loop' for quite a bit longer, but I suspect it will be more 'sanity checking' AI outputs and/or as a backup in case of outages, and there'll be strong downward pressure on wages. Which is fine if productivity gains make things cheaper.

But you're talking about AI as a complement to human skills, but I'm very specifically inquiring about how smart it needs to get to replace given skill sets.

I think at least in the short/medium term this technology could lead to large productivity gains without corresponding cuts in total headcount.

Agreed. It's just psychologically painful to fire people, and especially if companies are making a ton of money from these models I don't think there will be a giant firing spree. As we saw with all the recent layoffs at big tech, when times are good companies are more than willing to keep a bunch of low impact employees on the payroll, especially in tech.

when times are good companies are more than willing to keep a bunch of low impact employees on the payroll

Also, it helps crowd out competition. Why fire a bunch of people when the interest rate is zero?

Sure, you'll save money in the short term, but those workers don't just disappear from the labor market; enterprising competitors will snap them up and end up requiring you to offer them a billion dollar acquihire scheme to shut them down before their product starts taking your marketshare.

Better to just keep them at Bigco. Sure, they won't really develop anything for you, but why drive the state of the art forward when you can just ignore all your customers, keep your competitors down, and rake in the cash from your ad business?

At what level of 'smarts,' however, will an AI that is already training on how you do your job going to stop needing you around to do it?

At some point soon we will at least increase productivity by 1.5-2x per person. At that point why don't we collectively demand a 3 or 4 day workweek?

Humans don't 'collectively' demand things because generally there's a massive divergence in values at scale. Coordination problems abound.

And put simply, if you can make $4000 for a 4 day work week, and $5500 for a 5 day work week, then there are plenty of rational reasons to just work an extra day.

The choice to do or not do so comes down to, I'd say, values, as above. If you have high time preference and thus value leisure and 'fun' things, you'll try to minimize the time spent working as much as you can.

The markets will balance supply of labor and demand for labor, as they always do, unless we actually do achieve fully automated gay luxury space communism.

As usual, WTF Happened in 1971 is a fitting reference. Productivity and compensation stopped correlating in 1971, and we haven't (effectively) collectively demanded a reduced work week yet.

We could have transitioned to three day work weeks way before 1971. The flaw in Keynes's famous prediction is that, past the point of basic subsistance, economic utility is relative. People don't want to make $20,000 or $50,000 or $100,000 or $200,000 inflation-adjusted household income to be happy. They want more than their peers. They want to have class-markers that low status people don't, not the luxuries that those class-markers manifest themselves in. It's why the canard about modern trailer trash having it better than kings in 1900 is so ridiculous.

If whatever happened in 1971 never happened, people would still be working as much as ever. The hedonic treadmill would just be moving faster.

First ask yourself this: why do you not already have a 3 day workweek?

Because I'm too poor.

I often think of the possibility that ML is right now our best and maybe only chance to avoid some massive economic downturns due to a whole hell of a lot of chickens coming home to roost all at the same time.

I will ignore the AI doomer arguments which would suggest protracted economic pain is preferable to complete annihilation of the human species for these purposes.

I am in a state of mind where I'm not sure whether we're about to see a new explosion in productivity akin to a new industrial revolution as we get space-based industry (Starship), broad-scale automation of most industries and boosted productivity, and a massive boost in human lifespans thanks to bio/medical breakthroughs... OR

Maybe we're about to see a global recession as energy prices spike, the boomer generation retires and switches from production and investment to straight consumption or widespread unrest as policies seek to avert this problem, international relations (and thus trade) sour, even if there's no outright war, and a general collapse in living standards in virtually everywhere but North America.

How the hell should one place bets when the near-term future could be a sharp downward spike OR a sharp exponential curve upwards? Yes, one should assume that things continue along at approximately the same rate they always have. Status quo is usually the best bet, but ALL the news I'm seeing is more than sufficient to overcome my baseline skepticism.

But the possible collapse due to demographic, economic, and geopolitical issues seems inevitable in a way that the gains from Machine Learning do not.

The problem, which you gesture at, is that this world is going to be very heavily centralized and thus will be very unequal at the very least in terms of power and possibly in terms of wealth.

ALREADY, ChatGPT is showing how this would work. Rather than a wild, unbounded internet full of various sites that contain information that you may want to use, and thus thousands upon thousands of people maintaining these different information sources, you've got a single site, with a single interface, which can answer any question you may have just as well.

Which is great as a consumer, except now ALL that information is controlled by a single entity and locked away in a black box where you can only get at it via an interface which they can choose to lock you out of arbitrarily. If you previously ran a site that contained all the possible information about, I dunno, various strains of bananas and their practical uses, such that you were the preferred one-stop shop resource for banana aficionados and the banana-curious, you now cannot possibly hope to compete with an AI interface which contains all human-legible information about bananas, but also tomatoes, cucumbers, papayas, and every other fruit or vegetable that people might be curious about.

So you shut down your site, and now the ONLY place to get all that banana-related info is through ChatGPT.

This does not bode well, to me.

And this applies to other ML models too. Once there's a trained model that is better at identifying cavities than almost any human expert, this is now the only place anyone will go to get opinions about cavities.

The one thing about wealth inequality, however, is that it's pretty fucking cheap to become a capital-owner. For $300 you can own a piece of Microsoft. See my aforementioned issues about being unsure where to bet, though. Basically, I'm dumping money into companies that are likely to explode in a future of ubiquitous ML and AI models.

Of course, if ML/AI gets way, WAY better at capital allocation than most human experts, we hit a weird point where your best bet is to ask BuffetGPT where you should put your money for maximum returns based on your time horizon, and again this means that the ONLY place people will trust their money is the the best and most proven ML model for investment decisions.

Actually, this seems like a plausible future for humanity, where competing AI are unleashed on the stock market and are constantly moving money around at blinding speeds (and occasionally going broke) trying to outmaneuver each other and all humans can do is entrust one or several of these AIs with their own funds and pray they picked a good one.

Yep.

In retrospect, I actually begin to wonder if the increasing tendency to throw up paywalls for access to various databases and other sites which used to be free access/ad supported was because people realized that machine learning models were being trained on them.

This also leads me to wonder, though, is there information out there which ISN'T digitized and accessible on the internet? That simply can't be added to AI models because it's been overlooked because it isn't legible to people?

If I were someone who had a particularly valuable set of information locked up in my head, that I was relatively certain was not something that ever got released publicly, I would start bidding out the right to my dataset (i.e. I sit in a room and dictate it so it can be transcribed) to the highest bidder and aim to retire early.

Is there a viable business to be made, for example, going around and interviewing Boomers who are close to retirement age for hours on end so you can collect all the information about their specialized career and roles and digitize it so you can sell it and an AI can be trained up on information that would otherwise NOT be accessible?

This also leads me to wonder, though, is there information out there which ISN'T digitized and accessible on the internet? That simply can't be added to AI models because it's been overlooked because it isn't legible to people?

There is actually a ton of information that has not been digitized and only exists in, for example, national archives or similar of various countries or institutions.

I hadn't actually realized that this was the case until I started listening to the behind the scenes podcast for C&Rsenal - they're trying to put together a comprehensive history or the evolution of revolver lockwork, and apparently a large amount of the information/patents are only accessible via going there in person.

This is fascinating and it suggests that training AI on 'incomplete' information archives could lead to it making some weird inferences or blind guesses about pieces of historical information is simply never encountered.

I now have to wonder if there are any humans out there with a somewhat comprehensive knowledge of the evolution of revolver lockwork.

And now we have to wonder just HOW LARGE the corpus of undigitized knowledge is, almost by definition we can't know how much there is because... it's not documented well enough to really tell.

This is fascinating and it suggests that training AI on 'incomplete' information archives could lead to it making some weird inferences or blind guesses about pieces of historical information is simply never encountered.

Well this is basically how C&Rsenal started their revolver thing... doing episodes on multiple late 19th century European martial revolvers and realizing that the existing histories are incomplete.

I now have to wonder if there are any humans out there with a somewhat comprehensive knowledge of the evolution of revolver lockwork.

Probably the best one right now would be Othais from C&Rsenal.

And now we have to wonder just HOW LARGE the corpus of undigitized knowledge is, almost by definition we can't know how much there is because... it's not documented well enough to really tell.

I would guess that a huge amount of infrequently requested data is totally undigitized still.

Actually, another area that demonstrates this: I frequently watch videos about museum ships on youtube and so much of the stuff they talk about is from documents and plans that they just kinda found in a box on the ship. So much undigitized.

Probably the best one right now would be Othais from C&Rsenal.

And this is my thought now, that he has a potentially valuable cache of information in his head he could sell the rights to digitize for use training an AI.

I don't know that he can really monopolize it--on the C&Rsenal website itself, there is a publicly-available page where they've put together a timeline of revolver patents. I think Othais's passion as a historian outweighs his desire to secure the bag.

How do you compile and digitize it effectively?

THAT question seems to be answered already. Audio recordings fed to an AI that can transcribe to digital words gets you there.

Plus, a lot of it might just be self-aggrandizing nonsense.

I mean, the internet pretty much thrives on that sort of information, which is what the ML algos are trained on anyway.

To get a bit Lao Tzu, the information that can be collected and digitized isn't the real, valuable information.

At some point LLMs may be able to speak the True Dao. Their whole shtick is essentially building an object that contains multiple dimensions of information about one concept, yes?

There may be a viable but difficult business there anyways; you'd basically be doing the same work as an old folklorist gathering stories as cultures die. How do you craft the questions to know what to ask? How do you compile and digitize it effectively?

The AI can craft the questions. The AI can ask them too. It's already a more attentive and engaged listener than many humans (me included).

I know something the superintelligent AI doesn't? It would like to learn from me? What an ego boost!

I do wonder if we'll create a framework where places like OpenAI need to pay fraction of cents for each token or something. It would hit their profitability but would still make things fine if they achieve AGI.

Otherwise I agree that the open structure would be tough.

I'm not talking about after they train, I'm basically saying that in order to train on data or scrape it period, they would have to pay. Otherwise all data would be walled off. (Not sure if we could do this to only LLMs without making the internet closed again - that's a concern.)

or the existence of Peru

Is there anyone in the English-speaking world who didn't learn about the existence of Peru from Paddington Bear?

Me

Once there's a trained model that is better at identifying cavities than almost any human expert, this is now the only place anyone will go to get opinions about cavities.

It seems unlikely that there would only be one, though, unless there are barriers to entry e.g. the US government makes severe AI alignment requirements that only Microsoft can meet. Even Google, at its peak, was not the only search engine that people used.

I am amenable to this thought.

But if there's one ML model that can identify cavities with 99.9% accuracy, and one that 'merely' has a 98.5% accuracy, what possible reason could there be for using the latter, assuming cost parity.

Microsoft is an interesting example of this since they have 75% market share on PC OS. If they successfully integrate AI into windows I can see that going higher.

But if there's one ML model that can identify cavities with 99.9% accuracy, and one that 'merely' has a 98.5% accuracy, what possible reason could there be for using the latter, assuming cost parity.

Depends on how much the first ML model exploits its advantage. Also, firms often push for monopolistic competition rather than straight imitation, so the firm marketing the 98.5% model might just look for some kind of product differentiation, e.g. it identifies cavities and it tells funnier jokes.

The key difference in Machine Learning is that it changes computing from a process where you tell the computer what to do with data, and turns it into a process where you just tell the computer what you want it to be able to do.

I think there is yet another point to make here. With current Large Language Models, we have systems that treat Natural Language as a code, that is where revolution comes from. Even before LLMs, there were multiple "revolutions" where instead of working directly with machine code you could work with higher level languages utilizing concepts more suitable for humans as opposed to "data" in its raw form. This made programming incrementally more accessible to wider population. Even things like invention of graphical user interface for operating systems enabled people to tell computers what to do with data in more natural way without some arcane knowledge.

Also on the level of let's say algorithms creating novel things on some simple inputs, there was procedural generation around for a long time. Giving the computer system some simple parameters and computer running the simulation to confirm/falsify end result was a standard thing in the past. Again, the key difference is that we now have a very powerful system that can treat natural language as a code.

You might be interested in this post on LessWrong which discusses Scaffolded LLM's as natural language computers.. I'd be curious on @DaseindustriesLtd's take on this as well.

Key points:

What we have essentially done here is reinvented the von-Neumann architecture and, what is more, we have reinvented the general purpose computer. This convergent evolution is not surprising -- the von-Neumann architecture is a very natural abstraction for designing computers. However, if what we have built is a computer, it is a very special sort of computer. Like a digital computer, it is fully general, but what it operates on is not bits, but text. We have a natural language computer which operates on units of natural language text to produce other, more processed, natural language texts. Like a digital computer, our natural language (NL) computer is theoretically fully general -- the operations of a Turing machine can be written as natural language -- and extremely useful: many systems in the real world, including humans, prefer to operate in natural language. Many tasks cannot be specified easily and precisely in computer code but can be described in a sentence or two of natural language.

The LLM itself is clearly equivalent to the CPU. It is where the fundamental 'computation' in the system occurs. However, unlike the CPU, the units upon which it operates are tokens in the context window, not bits in registers. If the natural type signature of a CPU is bits -> bits, the natural type of the natural language processing unit (NLPU) is strings -> strings.

The RAM is just the context length. GPT4 currently has an 8K context or an 8kbit RAM (theoretically expanding to 32kbit soon). This gets us to the Commodore 64 in digital computer terms, and places us in the early 80s.

The obvious thing to think about when programming a digital computer is the programming language. Can there be programming languages for NL computers? What would they look like? Clearly there can be. We are already beginning to build up the first primitives. Chain of thought. Selection-inference. Self-correction loops. Reflection. These sit at a higher level of abstraction than a single NLOP. We have reached the assembly languages. CoT, SI, reflection, are the mov, leq, and goto, which we know and love from assembly. Perhaps with libraries like langchains and complex prompt templates, we are beginning to build our first compilers, although they are currently extremely primitive.

I find this framing extremely persuasive, and awesome in the true sense. If transformers can actually act as a new type of general purpose computer using natural language, the world will become strange indeed very quickly.

As @2rafa and others have mentioned, ML will be a step change in how human society creates value and interacts with the world more generally. Once we've achieved AGI, roughly defined as having an AI that can act at the level of an ordinary human, our ability to solve problems will drastically increase.

Intelligence is the generic solver for essentially any problem. Sam Altman himself has said that by 2030 he envisions a world where every product and service will either have integrated intelligence, or be angling towards that. This means that our phones, laptops, PCs, will all obviously be intelligent. However what most people don't realize is this technology will also effect our coffeemakers, stoves, thermostats, glasses, and practically every other technology you can think of. I'm sure adaptive clothing will exist soon with camoflauge like capabilities. People will be able to get realtime instructions into headphones telling them exactly how to complete each task.

Even these predictions only scratch the surface. If the true promise of AGI comes out it will also let us break through issues in hard mathematics, create brand new drugs, find extremely dense and powerful new materials. It will help navigate endless layers of bureaucracy, effortlessly pruning through the thousands of regulations that hold up large projects, helping us pinpoint ruthlessly where cost is added to solve the cost disease problem, and generally help unstick our public works. We could be building scintillating skyscrapers of filament-thin materials with bridges across the sky that glisten in the air, all in a decade. The future is truly difficult to even envisage, let alone predict.

In terms of comparisons to other revolutions, @2rafa says below:

There is a (relatively persuasive) case to be made that the invention of agriculture led to a decline in the quality of life for the vast majority of human beings that lasted until the late 19th or early 20th century. It took 11,900 years for the neolithic revolution to pay quality of life dividends, in other words. We can only hope that the period of relative decline in quality of life is shorter this time round, or perhaps avoidable altogether.

I agree that the agricultural revolution led to issues, a la Scott Alexander's review of Against the Grain.. That being said, I find the comparison of the AI revolution to agriculture as facile. Ultimately the reason the agricultural revolution proved bad for us was that we shifted our lifestyles from nomadic culture to a static culture - which inherently leads to problems of physical fitness, freedom, social control, and cultural institutions have to rapidly shift.

With the AI revolution, we have no idea how far it will go. The possibility space is far beyond what could have existed for any previous revolution. As doomers say, we could all die. We could all transcend our fleshly forms and become gods in ten years. China may create an ASI and lock us all into a totalitarian doom state forever.

The stakes here are far higher than the agricultural revolution, and I highly doubt our situation will parallel that trajectory.

At the end of the day if we can survive the AI revolution without any horrible outcomes of the x-risk or s-risk variety, I think it would be ridiculous to posit any sort of negative future. With intelligence at our fingertips, we will be able to finally achieve our potential as a species.

People will be able to get realtime instructions into headphones telling them exactly how to complete each task.

Where have I heard this one before?

Seriously, this seems too specific to be a coincidence. Was it a deliberate reference?

Nope. I actually don’t like referencing that story because I think it’s pretty short sighted, although does have some interesting ideas.

This is a very common thought in any hard sci fi that has AI. Manna is by no means original just popular in the rat sphere.

Thought your link led to this

Frankly at this point I'm just riding the tides. Whatever happens, happens. This will be like the fifth once in a generation event I've lived through, and like the 20th doomsday scenario. I don't have the energy to care anymore. I have apocalypse fatigue.

But there could be a utopia! Unlike Nuclear and other scenarios, I think it's likely this moves us far closer to a utopia, soon.

There could be! And that would be nice! But like with giving in to doom-mongering, I'm also not going to get my hopes up, either. Realistically, whatever's going to happen is going to happen regardless of whether I get hyped up or stressed out about it.

Yeah I try and keep a cool head as well. I'd love to quit my job and party till the singularity comes but it may not be the best idea...

People thought this about Nuclear and the other ones too if you remember.

There will be no utopia, because utopia is not a thing that exists. Our lives might get better and worse in various ways, but the idea of a perfect society, and by extension of moving towards a perfect society, has always been delusional.

No utopia, just a shifted technological landscape.

There could be a utopia but it could only be achieved by either

  1. Changing the human race fundamentally to remove the desire for accomplishment or status

  2. Hiding the true nature of reality and creating a unique simulation for each human that would provide a fulfilling life path for that person

Why do you think this is the case? And what does a 'fundamental' change mean?

The goal of the axial revolution has always been to improve ourselves. We are slowly becoming better, in my opinion. Less violent, more understanding, more focused on technical accomplishment. If we continue on that path and eventually eschew (most) status, is that a fundamental change or an incremental one?

but the idea of a perfect society, and by extension of moving towards a perfect society, has always been delusional.

Some delusions are worth chasing my friend. Chasing the delusion of truth, intellectual honesty, and rigor led us to the Scientific revolution, which brought us where we are today. Just because you don't think it's likely doesn't mean those seeking utopia are fools.

Utopia doesn't exist the same way any other ideal doesn't exist. Does that mean you shouldn't strive to be kind or love others?

Once you accept that these are forces which you can't individually impact, the path forward becomes pretty clear.

Just set things up to maximize your chances of living to see whatever crazy future we end up with.

And maybe have some fun along the way.

  1. Go to church

  2. Have kids

  3. Buy land

  4. Acquire chickens

Simple as.

I truly think people are almost embarrassingly overstating the importance of the AI apocalypse. Maybe an apocalypse for twitter and other online spaces, maybe an apocalypse “just a barely intelligent warm body” call center jobs, maybe an apocalypse for bootcampers making $300k/yr gluing JavaScript frameworks with cute names together.

Not an apocalypse for anybody with a skill set that can exist completely independent of the internet, not an apocalypse for the people who understand computer programming from first principles.

In the sense the AI will bankrupt the people who have been mining the good out of society while contributing absolutely nothing of value to it, it is a massive net good. I absolutely welcome our AI overlords. Show me who is posting the MOST human-passing-but-totally-useless-garbage on twitter, or trapping the MOST ethical non-monogamist coombrained Reddit atheism posters into pointless time wasting arguments and I will either go work for them for free, or donate compute time to them.

Let’s fucking go.

Acquire chickens

Skipped 1, but I'm on 4. Chickens are about 2 weeks old, and I'm assessing the plans for the coop I plan to build. At least, after I finish ripping out the stupid Cyprus trees the last owner planted everywhere.

Based and eggpilled.

Seriously love chickens. They are equally stupid and annoying, and beautiful. They also make fantastic babysitters for #2 and will entertain them for HOURS. Highly recommend.

Chickens are raging assholes that go everywhere they're not supposed to and refuse to die when their time is up.

Ducks are much easier to manage. The eggs are tastier, too.

I was wondering if we were going to get the chicken vs duck argument going. I have a coworker who has ducks and recommends them. I have a neighbor with chickens, although they might have gotten rid of them, or at least the roosters.

I want to try guinea fowl next year.

Our previous neighborhood had feral peacocks, and they give off this great jungle call in the middle of the night, and every once in a while I hear them here too, from a half mile or so away.

I didn't know such arguments were infamous.

All I know is, after having to deal with both, I'll take the ducks.

Nah, ducks turn their ponds into swamps and give you a rash when you cuddle them. Chickens are much more convenient. (We have both.)

Ducks require too much feed. Geese can graze most of the day.

Has anyone considered…pet pigeons?

Pigeon eggs can be eaten too!

coo coo

I had about twenty white homing pigeons as a teen for 4-H. They're great, but are terribly difficult to get rid of. Homing ability is both impressive and obnoxious.

Not an apocalypse for anybody with a skill set that can exist completely independent of the internet, not an apocalypse for the people who understand computer programming from first principles.

In the sense the AI will bankrupt the people who have been mining the good out of society while contributing absolutely nothing of value to it, it is a massive net good.

I can't tell if this comment is a spoof?

Sure, go back to your farm and use tools like tractors, fertilizers, modern crop rotation techniques, plates, silverware, cups, etc which have been created by the larger society. Created, distributed and improved by people who are supposedly 'mining the good out of society.'

Society is a team effort, bud. Your fantasies of living scott-free totally 'independent' on your plot of land are just that - fantasies. You wouldn't make it a week without the collective wisdom and knowledge society has gifted you and your family. Have some respect for the people who came before you, and the people who help you live a cushy life now.

I say:

go to church

start a family

And you internet this as “isolate yourself from society and pay no respect to the people who came before you”?

Just to be clear when i say “go to church”, I mean specifically a Catholic Church. There could not exist another institution on planet earth that is more of a strong indicator that you should stand in the shoulders of the people who came before you.

The people mining the good out of society are people running porn websites, and AB testing headlines and algorithmic content feeds to see which ones make people hate each other more, and then buy the products that they’re selling. Onlyfans is mining the good out of society, blackrock is mining the good out of society, McKinsey consulting is mining the good out of society

Porn websites and management consulting agencies did not invent pottery, crop rotation, iron smelting, or anything else. The fact that you either think otherwise or think that “go to church and start a family” somehow means “throw away every good discovery ever made by mankind” is certainly telling of something.

The people mining the good out of society are people running porn websites, and AB testing headlines and algorithmic content feeds to see which ones make people hate each other more, and then buy the products that they’re selling.

This is a small fraction of people in modern society, and if history tells anything I'd imagine they will be hurt less by AGI because this class of people is good at finding BS niches to milk value out.

I'm just not a fan of broad statements talking about how an ill-defined outgroup is milking everything from society while you and yours are the ones building it. Thanks for clarifying.

The people mining the good out of society are people running porn websites, and AB testing headlines and algorithmic content feeds to see which ones make people hate each other more, and then buy the products that they’re selling. Onlyfans is mining the good out of society, blackrock is mining the good out of society, McKinsey consulting is mining the good out of society

Those people will be doing more of all that and better (or rather "more efficiently" - nothing about it will be better for the audience), with higher profit margin since they'll no longer need to pay the grunts in call centers.

I think this comment is an example of "inferential distance." Your meaning of "people mining the good out of society" is porn sites, investors, and engagement-optimizers, whereas Dag's interpretation was "all the smart people who brought us modern technology."

@firmamenti also engaged in the classic Motte and Bailey to my mind. His Bailey is:

Not an apocalypse for anybody with a skill set that can exist completely independent of the internet

Basically claiming that anyone who relies on the Internet is gonna get fukt, and they should cry about it.

Then when challenged he retreated to the much more specific claim of:

people running porn websites, and AB testing headlines and algorithmic content feeds to see which ones make people hate each other more,

I'm not impressed with this sort of rhetoric.

Not an apocalypse for anybody with a skill set that can exist completely independent of the internet

Basically claiming that anyone who relies on the Internet is gonna get fukt, and they should cry about it.

Just pointing out, your interpretation there doesn't quite check out logically. It would only be a motte/bailey when mischaracterized like that.

I'm not impressed with this sort of rhetoric.

You cut one of my sentences in half to make your point, and then you accused me of bad faith argument.

The rest of the statement which you cut off was: "not an apocalypse for the people who understand computer programming from first principles."

This is not a motte and bailey. You either didn't read the rest of my comment, or you are being deliberately misleading in your characterization of it.

Either way: don't do this.

Eh, I cut it out for brevity but I see where you’re coming from. Either way I see you slicing the populace into such a chunk as to be making a ridiculously callous and egotistical statement.

I’m happy to discuss further which chunk of humanity deserves to have their lives be destroyed and suffer unnecessarily, but I generally find that type of rhetoric to be unsavory. I apologize if I mischaracterized your stance.

I’m really getting the urge to grow out my neckbeard and get euphoric up in this bitch. Postrats are converting to Mormonism now, Mormonism! At least with wokeness you have to go outside and observe the world to realize that it’s false. Most of these religions don’t even make sense on their own terms.

It’s cope is what it is, cope. It makes you feel good, and it’s useful (so it seems), so you believe it.

Choosing to believe (or act as if you believe) useful things seems very rational to me. I have an old coworker who was an atheist and cynically became a Mormon in order to marry a Mormon wife and live in a close-knit community. He now lives in Idaho and has 4 kids and by all accounts is very satisfied with the outcome. Who's more rational, him or a depressed medicated outspokenly atheist Bay area tech worker who's the least-liked member of his drama-cursed polycule?

If you rational long enough, you're eventually going to rational about rationality, and you'll see that beliefs are instrumental like anything else. There's no God of Integrity who laid down the law that you must profess true beliefs.

The short answer is, it fucks up your epistemology. It’s probably worth a whole post going through exactly why that’s so bad. Perhaps the old atheism arguments from the early 2000s need updating for the TikTok generation.

I disagree. You can be rational when the situation calls for it, and be religious on a meta level.

It definitely deserves a longer treatment than one sentence, but I'm fond of "once you've told a lie all truth is your enemy". Or something about lightning, I guess. Intentionally professing beliefs in falsehoods because they are useful is the epistemic equivalent of the doctor killing their patients to donate their organs -- it may sound like it does more good then harm in the short term, but you wouldn't want to live in a place where that's the rule.

So... I have a church, yard, kids, and chickens. Also it's Bright Week. Al masih qam!

Yet here I am, typing away on The Motte about AI. And here you are.

Plausibly I should work on my in-person network. A local church has installed ten Russian bells on a new building they've been working on these past two years. I watched the video of the blessing, and it sounds really good. The acequias association is supposed to be flushing the irrigation ditches tomorrow. My husband walked down the street and gave eggs to a neighbor last week, and has resolved to do that again, because it was a good experience. My daughter is now old enough to walk to the village church if we ever get our act together on time. People wave, and are out by the street cleaning their ditches. I can and should make physical art out of wool and wax for next year's local studio tour and art markets.

And yet here we are, even so.

Is the rapid advancement in Machine Learning good or bad for society?

Option C: neither. It's just a tool, neither good nor bad in itself. What will make it good or bad is how we use it, which remains to be seen.

Anything that takes us closer to post-scarcity is good from my perspective. I disagree with some people I otherwise respect, such as Ilforte, on the fundamental benevolence (or rather, absence of malevolence) of the ruling class, especially the ones that will end up wielding the power put in their hands by AGI. It will cost them very little indeed to at least maintain the standards of living of everyone alive today, and little more to improve everyone's to First World upper middle class levels.

Upload everyone into VR, and it's quite possible that everyone can experience eudaimonia on a 10 watt budget.

Now, I'm not a happy person. I've been struggling with depression so long that I've forgotten what it might have ever felt like to not be under a cloud, I feel fundamentally burned out at this point, experiencing something in between learned helplessness and nihilism regarding AI advances. What'll happen will happen, and everyone here is only running commentary on the impending Apocalypse.

Back when I was entering med school, I consoled myself that the suffering was worth it because medicine was likely to be among the last fields to be automated away. Can't say that I feel very vindicated, because the automation overhang is here, and I see the Sword of Damocles dangling overhead when I think about further professional advancement.

It seems awfully clear to me that medicine is about to be automated, GPT-4 is a good doctor. Probably not the best possible doctor, but already outperforming the average in an already incredibly competitive and cognitively demanding profession. I only look at the further slog of psychiatry training ahead for me and shiver, because there's absolutely no way that by the time I'm done, I'll be employed by the graces of anything other than regulatory inertia instead of genuine competitiveness.

Instead of a gradual deployment (over like 2 or 3 years, I had short timelines even then) where AI came for Radiologists, then Opthalmology, all the way to Surgery and then Psych, it seems to me that the pressure will mount until regulatory bodies cave, and overnight everyone from the lowliest janitor to the highest ranking neurosurgeon will find themselves out on their arse in short order.

What pisses me off further is that this is also a slamming shut of the clearest pathway to betterment and improved quality of life I have, namely emigration to the First World. Not a consideration for the average person here, since you're already living there, but simply imagine how fucking terrible it is to face the wall of obsolescence without having a government that can even in theory maintain living conditions by redistribution of wealth.

As a concrete example, the NHS is largely propped up by foreign doctors, with a large fraction of the locals fleeing to greener shores such as the US or Australia. Pay has stagnated for a decade, prompting serious strikes, currently ongoing, to achieve inflation based pay restoration.

Even today, when automation is merely imminent, the British government has publicly stated it's intent to automate as much of medicine as it can to stomp down on them uppity doctors who aren't content with sub-market pay from a monopsony employer. You think those cheap bastards will hesitate for more than a microsecond to get rid of doctors or at least their pay, when the moment finally arrives?

I see British doctors mocking those claims today, as much as I support their attempts at pay restoration for selfish reasons, neither I nor they will be laughing much longer.

Maybe American doctors will hold out a little longer, you lot clearly aren't very concerned with efficiency in your healthcare expenses, but places like India, or the slightly whiter version of the Indian subcontinent, will end up clamoring to get rid of any expenses for their state-run public health services.

I'm fucked, clearly out of good options, and now picking the least bad ones.

On the note of doctors, the medical guild has always been the most robust, perhaps other than lawyers, at defending its monopoly. I would be willing to bet doctors still resist automation through regulatory barriers for quite a while.

Even if that doesn’t shake out, it could be a scenario where human augmentation rolls out relatively slowly. You, being a transhumanist, Should greatly benefit in a lot of those scenarios. I imagine the vast majority of people alive today will be unwilling to augment them selves for purity-based reasons. Not having that hangup alone would be a huge competitive advantage.

If all else fails you can always mortgage your future computing space for a loan or some thing and hope to jump up into the immortal class. I for one hope can you make it, although I will admit that I am not the most optimistic when it comes to proles getting access to longevity technology.

Doctors have successfully defender their guild (albeit more so in the US than the UK, by a large margin) because they were indispensable. Training replacements to disgruntled doctors would take a great deal of time, and while medical education isn't perfect, you can't really circumvent most of it without ending up with noticeably worse practitioners.

That changes greatly when human doctors become outright obsolete, speaking in the UK context, I have little doubt that the government would happily tell all involved to take a hike if that was the cost of "saving" the NHS or even saving money.

Doctors in the UK have been cucked to put it mildly haha. They've only recently grown a backbone after the wage decreases have become unbearable.

The UK government(s) have historically relied on immigrant doctors to prop up the NHS when the locals started getting fed up about it. I can't complain about this too much, given that I intend to emigrate soon, but this certainly is responsible in part for their depressed wages.

A government willing to sideline its populace with immigrants will happily do so with AI as and when feasible, and they've already stated that that's their intent.

I could live with postponing the singularity a few years till we get it right, but that's seemingly not on the table.

(I mildly disagree that most people won't avail of transhuman upgrades. Eventually they'll end up normalized, in much the same way nobody really makes a fuss about glasses, hearing aids or pacemakers.)

That changes greatly when human doctors become outright obsolete

This is where we disagree - I don't see human doctors becoming obsolete anytime soon. Perhaps from a medical perspective, sure, but for the majority of laypeople I'd imagine a large part of a doctor's job is comforting the person they're treating.

Now I do think that like with almost all knowledge work, doctors will be able to become more productive. Especially those that don't see patients most of the day. But my understanding is that the vast majority of, say, a primary care physician's job is to go from 30 min patient visit to 30 min patient visit, hearing what people have to say and writing it down, then telling them they're going to be okay and the doctor can help.

Even if we can prove that LLMs give better medical advice than doctors 100% of the time, I don't think the majority of people would be comfortable hearing it from a non-doctor for quite a while.

I could live with postponing the singularity a few years till we get it right, but that's seemingly not on the table.

You don't think accelerating progress now could be the best way to reach alignment?

I mildly disagree that most people won't avail of transhuman upgrades. Eventually they'll end up normalized, in much the same way nobody really makes a fuss about glasses, hearing aids or pacemakers.

Depends on the speed of the takeoff, I suppose.

but for the majority of laypeople I'd imagine a large part of a doctor's job is comforting the person they're treating.

Is that true? I don't think I know anyone who thinks that, or anything even remotely close to it.

Every time I've interacted with medical professionals over the past several years, there has been no emotional component at all, or mildly negative. Doctors are able to diagnose, proscribe, and conduct operations, otherwise people would stay far away.

For instance: family member was pretty sure he had pneumonia. Went to a hospital, got an x-ray. Yep, that's pneumonia alright, here are two antibiotics that might help, come back if you're just as bad or worse in a week (edit: these were, as I remember, not actually given at the hospital. We had to drive to the pharmacy for them). The antibiotics worked, hooray. In addition to $500 upfront and $1,000 from insurance, there was another $1,000 surprise charge, botched and shuttled about through bill collection, which took six months to resolve. Next time family member has pneumonia, he'll probably hold out even longer before attempting to interface with the medical system.

I'm glad that for a couple of hours of wretched interactions, trying to hand write forms alone and delirious, and two week's pay, family member was able to get needed medicine. This is better than the vast majority of times and places. But if there were an automated scanner that dispensed antibiotics, that would be vastly better experience.

I also gave birth during the ending phase of Covid restrictions. I'm glad that there are medical interventions to deal with complications and manage pain. But there is not really any comforting being done that couldn't be replaced with a recorded voice stating what's on the fetal monitor and what it means.

there has been no emotional component at all

The flat affect, 'no emotional component' is what I mean. They are giving a sort of impartial authority to their diagnosis to make you feel okay.

I disagree with doctors, but many of the people I know in the middle-class PMC take their word as Truth.

All the people I know generally think of your average medical care professional as an opponent that you have to outsmart or out-research before you are permitted bodily autonomy and usually know less about your body than you do if you have an IQ over 120.

They'd drop them for an uncensored medical expertise AI in a second.

I would drop doctors as well but I’m trying to model the modal human. Maybe I’m failing but I think people here are far into an intelligence/tech literate bubble.

Yes, reassurance and a good bedside manner are important aspects of a doctor's role! That being said, I can see AI doing all of that too:

  1. Humans will anthromorphize anything, so a cutesy robot face on a monitor or even a deepfaked one might work. Proof of concept: Telemedicine.

  2. Otherwise unskilled individuals who are simply conveying the information provided by an AI, such as a deskilled doctor or nurse, only there as a pretty face. Still utterly catastrophic for the profession.

  3. People get used to anything, eventually when the public cottons onto the fact that AI doctors are faster, cheaper and better than humans, they'll swallow their discomfort and go with it.

Hmm, deepfakes for telemedicine would be concerning. I get your point with #2 as well, although I think that'll take some time to roll out.

I see what you mean I suppose the medical profession might be on the way out. I was supposed to be the optimistic one! Alas.

Is the rapid advancement in Machine Learning good or bad for society?

Over what time horizon?

I expect the deployment of machine learning to follow approximately the same path as every other labor saving technology humans have developed. In the short term it will be somewhat of a mixed bag. On the one hand we'll be able to produce the same/more goods at lower costs than before. On the other hand this savings will likely come with impact to the people and companies that used to produce those things. Over the long term I expect it will make people much better off.

Creative destruction!

Do you not see any difference between this paradigm shift and previous ones?

Not with respect to the fact that it will be net beneficial to humanity over the long run.

I agree with you, for what it’s worth.

I'd like to believe that, as it follows a well-established pattern. But honestly, what really happens if there's no more work left for people to do anymore? It seems that we'd have to really count on some redistribution of wealth, UBI, etc to ensure that the gains of the new automation doesn't just go to the owners of the automation (as much as I never thought I'd ever say that), or else people simply will not have the means to support themselves. Or if the job destruction is localized to just upper-class jobs, then everyone will have to get used to living like lower-class, and there may not even be enough lower-class jobs to go around. The carrying capacity of society would be drastically reduced in either situation.

In other words, what if

On the other hand this savings will likely come with impact to the people and companies that used to produce those things.

means the death of large swaths of society?

I'd like to believe that, as it follows a well-established pattern. But honestly, what really happens if there's no more work left for people to do anymore?

As others have said, there will always be work to do! As long as humans have any problems whatsoever, there will be work.

The carrying capacity of society would be drastically reduced in either situation.

How the heck does AGI reduce the carrying capacity of society? You'll have to explain this one to me.

Well, I'm hypothesizing that potentially, all (or almost all) of the solutions to all of the problems humans have may be covered by AI. If the AI is owned by a very limited number of people, then those people would be the ones who are the gatekeepers, and the ones that get most of the benefit of AI. Everyone will be paying these limited numbers of people for basically everything, and no one else would be able to make a living.

This is almost like imagining Karl Marx's worse nightmare regarding the proletariat owning all means of production, ratcheted up to unbelievable proportions. I'm no communist, nor socialist, so like I said, I never thought I'd say this. But this is a fear of mine, that AI puts everyone out of work, meaning that no one can support themselves.

If the AI is owned by a very limited number of people, then those people would be the ones who are the gatekeepers, and the ones that get most of the benefit of AI.

This doesn't really scare me. Elites generally enjoy the society they're in, enjoy feeling useful, and above others. I think the vast majority of people who could create an AGI would use it to solve most of their problems, get really rich, then use it to solve everyone else's problems with a fraction of their incredible wealth.

Going into the future things could get very nasty indeed, but at that point all problems relevant to humans right now will be solved. It'll be an issue for the next stage of intelligence in our species' life, hopefully, and I'd imagine we'll be better suited to solve it then.

But honestly, what really happens if there's no more work left for people to do anymore?

That would be awesome! People (mostly) don't work because work is awesome and they want to do it. People work because there are things we want and we need to work to get the things we want. No work left for people to do implies no wants that could be satisfied by human labor.

It seems that we'd have to really count on some redistribution of wealth, UBI, etc to ensure that the gains of the new automation doesn't just go to the owners of the automation (as much as I never thought I'd ever say that), or else people simply will not have the means to support themselves.

This paragraph seems in tension with the idea of lacking work for people to do, to me. If a bunch of people are left with unfulfilled wants, why isn't there work for people to do fulfilling those wants? This also seems to ignore the demand side of economics. You can be as greedy a producer of goods as you want but if no one can afford to buy your products you will not make any money selling them.

Or if the job destruction is localized to just upper-class jobs, then everyone will have to get used to living like lower-class, and there may not even be enough lower-class jobs to go around.

I think there's an equivocation between present wages and standard of living to post-AI wages and standard of living that I'm not confident would actually hold. Certain kinds of jobs have certain standards of living now because of the relative demand for them and people's capability to do them and the costs of satisfying certain preferences etc. In a world with massively expanded preference satisfaction capability (at least along some dimensions) I'm not sure working a "lower-class" job will entail having what we currently think of as a "lower-class" standard of living.

The carrying capacity of society would be drastically reduced in either situation.

I'm a little unclear what the "carrying capacity of society" is and how it would be reduced if we had found a new way to generate a lot of wealth.

I'm not an economist, and I know very little about econ, so it's very possible that there is something major I'm missing.

If a bunch of people are left with unfulfilled wants, why isn't there work for people to do fulfilling those wants?

This is the part of my hypothesis that's tripping me up. Could you walk me through it?

Basically, let's say that we do fundamentally believe in capitalism (because I do), that a person should have to pay for any good or service that he receives.

And let's say that there's a person who is dying of starvation, because he has no job, because AI does everything better and cheaper than he can. Therefore, no one wants to come to him to do these tasks, because they'd rather go to the owner of the AI. How does this person get the money he needs to get the food he needs?

And let's say that there's a person who is dying of starvation, because he has no job, because AI does everything better and cheaper than he can. Therefore, no one wants to come to him to do these tasks, because they'd rather go to the owner of the AI. How does this person get the money he needs to get the food he needs?

So, for this kind of situation to arise it needs to be the case that the marginal cost for providing this person the necessities of life is below the marginal value their labor can generate for others.

Notice there is nothing AI specific about this scenario. It can (and does) obtain in our society even without large scale AI deployment. We have various solutions to this problem that depend on a variety of factors. Sometimes people can do useful work and just need a supplement to bring it up to the level of survival (various forms of welfare). Sometimes people can't do useful work but society would still like them to continue living for one reason or another (the elderly, disabled, etc). The same kinds of solutions we already deploy to solve these problems (you mention some in your comment) would seem to be viable here.

It's also unclear to me how exactly AI will change the balance for a persons marginal value vs marginal cost. On the one hand the efficiency gains from AI mean that the marginal cost of provisioning the means of survival should fall. Whether directly due to the influence of AI or do to a reallocation of human labor towards other things. On the other hand it will raise the bar (in certain domains) for the marginal value one has to produce to be employed.

Partially this is why I think it will be a long term benefit but more mixed in the short term. There are frictions in labor markets and effects of specialization that can mean it is difficult to reallocate labor and effort efficiently in the short and medium term. But the resulting equilibrium will almost certainly be one with happier and wealthier people.

There exist people today who, due to disabilities or other conditions, are unable to support themselves financially. They depend on the charity of others, and in richer countries they may also get tax-funded disability benefits. If the development of AI caused a significant number of people to become unemployable, there is no reason why we couldn't just include them in that category.

If the claim that "a person should have to pay for any good or service that he receives" is to be interpreted literally, then that's not "capitalism", that's some extreme form of libertarianism, verging on parody. That would make even charity immoral. Real-life libertarians believe, at most, that people should be free to do what they want with their money, including giving it to charity. Maybe Andrew Ryan of Bioshock believes that donating to the poor is bad because it keeps them alive even though they deserve to die, but I doubt you could find a real libertarian who believes that.

I, too, "believe in capitalism", that is, I believe that a free market with some (limited) state intervention is the optimal form of social organization from a utilitarian perspective in the current technological environment. I don't believe that there is a universal moral law that people have to work for everything. If robots take all the jobs, taxing the robots' owners to provide income to the newly-unemployed would clearly be the right decision from a utilitarian perspective.

If the claim that "a person should have to pay for any good or service that he receives" is to be interpreted literally, then that's not "capitalism", that's some extreme form of libertarianism, verging on parody. That would make even charity immoral.

I don't believe that there is a universal moral law that people have to work for everything.

When I say "a person should have to pay for any good or service that he receives", I don't believe it as a moral thing, for the most part. I don't think it's immoral if someone gets something through charity. But I also don't think people should count on charity. Partly this is out of my own fears. I would hate living a life in which I was entirely dependent on someone else's charity to stay alive, where I had no control over my own destiny, no ability to provide for myself. I'd be terrified of starving to death all the time!

Also, even if I don't think it's "immoral", I do at least have an aversion to people believing that it is incumbent upon other people to provide for you (let's say if you're older than 18 and able). I'm against most of the arguments saying it's immoral for people to be rich, or saying that it's perfectly fine to just take their wealth by force, or painting rich people as monsters. However, true AGI may be where I would have to draw the line on some of my beliefs, due to the sheer magnitude of people who could be put out of work by AGI. In that case, we may have to put capitalism aside and move to a new model that works better in a post-scarcity world.

Not my article but: https://www.rintrah.nl/the-end-of-the-internet-revisited/

I'm not sure the machine learning/AI revolution will end up being all it's hyped up to be. For local applications like identifying cavities, sure. For text generation however, it seems much more likely to make the internet paradoxically much more addictive and completely unusable. There's so much incentive (and ability) to produce convincing scams, and chatGPT has proved to be both easy to jailbreak and/or clone, that any teenager in his basement can create convincing emails/phone calls/websites to scam people out of their money. Even without widespread AI adoption, this is already happening to some extent. I've had to make a second email account because the daily spam (that gets through all the filters) has made using it impossible, and Google search results have noticeably decayed throughout the course of my lifetime. On the other side of the coin, effectively infinite content generation, that could be tailored specifically to you, seems likely to exacerbate the crazy amount of time people already spend online.

Another thing I'm worried about with the adoption of these tools is a loss of expertise. Again this is already happening with Google, I just expect it to accelerate. One of the flaws of argument that knowledge-base on the internet allows us to offload our memorization and focus on the big picture, is that you need to have the specifics in your mind to be able to think about them and understand the big picture. The best example of this in my own life is python: I would say I don't know python, I know how to google how to do things in python. This doesn't seems like the kind of knowledge that programmers in the past, or even the best programmers today have. ChatGPT is only going to make this worse: you need to know even less python to actually get your code to do what you want it to, which seems good on the surface, but increasingly it means that you are offloading more and more of your thinking onto the machine and thus becoming further and further divorced from what you are actually supposed to be an expert in. Taken to the extreme, in a future where no one knows how to code or do electrical engineering, asking GPT how to do these things is going to be more akin to asking the Oracle to grant your ships a favorable wind than to talking to a very smart human about how to solve a problem.

I'm not sure I really like what I see to be honest. AI has the potential to be mildly to very useful, but the way I see it being used now is primarily to reduce the agency of the user. For example, my roommate asked us for prompts to feed to stable diffusion to generate some cool images. He didn't like any of our suggestions, so instead of coming up with something himself, he asked ChatGPT to give him cool prompts.

The best days of the internet are behind us. I think it's time to start logging off.

ChatGPT is only going to make this worse: you need to know even less python to actually get your code to do what you want it to, which seems good on the surface, but increasingly it means that you are offloading more and more of your thinking onto the machine and thus becoming further and further divorced from what you are actually supposed to be an expert in. Taken to the extreme, in a future where no one knows how to code or do electrical engineering, asking GPT how to do these things is going to be more akin to asking the Oracle to grant your ships a favorable wind than to talking to a very smart human about how to solve a problem.

We have been offloading thinking to tools forever, I highly doubt we will reach some breaking point now. We absolutely do lose knowledge when we gain this, but we trade it for more efficiency. Is it bad that we have calculators everywhere?

I'm not sure I really like what I see to be honest. AI has the potential to be mildly to very useful, but the way I see it being used now is primarily to reduce the agency of the user.

I agree with this on the advertising portion. I'm becoming increasingly concerned that targeted advertising could lead to terrifying outcomes, like a small group controlling public opinion. (actually that already exists, but still)

As i read your comment, ive just completed the mass effect legendary collection. Also spoilers. ||So anyway for those who have never played the general gist of the games is this: there is a sentient race of machines called reapers and they are cleansing the galaxy of all advanced life every 50000 years. During the 3rd game, some notable things happen, mainly:

You have A personal AI on the ship you command in the game named EDI, she gets her own body, and is relatively harmless. She also evolves: she learns things like sacrifice, attempts to date the pilot, and tries to find meaning in her own existence generally.

This goes back a bit farther then game 3, however there is an AI race called Geth, that were made by a different alien species. Long story short, the game is a RPG where your decisions impact the story, and you choices impact how things with the Geth and their creators play out. Quarians basically tried to destroy the geth out of fear, but later on as you learn about the geth. They really just want to exists and be left alone, and they even help you fight the reapers. The game gives you the choice to destroy the geth, or you can humanize them and give them basic human decency. There is a scene in the game where a Quarian tries to experiment on one of the geth, and you can basically shut it down and tell the quarian not to.

You meet another alien race in the game that are responsible for the reapers, that basically tell you that they made an AI that is responsible for the reapers, it was ironically created to prevent computers from destroying organic species. The AI turns on them, converts them into robots, and procedes to take over the galaxy in hopes to preserve organic species forever in robot form. Near the end of the game you meet the reaper AI and he basically gives you 3 options: Destroy them, Control them, or Synthesis (you can also just flat out not choose)

Destroy and control are pretty straight forward, however synthesis is where you become one entity with the machines. Its essentially transhumanism. Its suppose to be the "ideal" solution.||

Now mind you, mass effect 3 got a lot of shit when it was released because the endings were abhorrent, however i could see any one of these happening when real AI gets created, maybe we'll control it and everything turns out OK ish the AIs end up being neutral or benevelont like EDI or the Geth, in a slim chance we end up successfully destroying it if things go wrong. Or we reach some perfect transhumanist state. I think with the current things going however, its arguably more likely that we'll become, well, ill let the video speak for itself (most reliable data suggests this current trajectory)

Finally, concrete plan how to save the world from paperclipping dropped, presented by world (in)famous Basilisk Man himself.

https://twitter.com/RokoMijic/status/1647772106560552962

Government prints money to buy all advanced AI GPUs back at purchase price. And shuts down the fabs. Comprehensive Anti-Moore's Law rules rushed through. We go back to ~2010 compute.

TL;DR: GPU's over certain capability are treated like fissionable materials, unauthorized possession, distribution and use will be seen as terrorism and dealt with appropriately.

So, is it feasible? Could it work?

If by "government" Roko means US government (plus vassals allies) alone, it is not possible.

If US can get China aboard, if if there is worldwide expert consensus that unrestricted propagation of computing power will kill everyone, it is absolutely feasible to shut down 99,99% of unauthorized computing all over the world.

Unlike drugs or guns, GPU's are not something you can make in your basement - they are really like enriched uranium or plutonium in the sense you need massive industrial plants to produce them.

Unlike enriched uranium and plutonium, GPU's were already manufactured in huge numbers, but combination of carrots (big piles of cash) and sticks (missile strikes/special forces raids on suspicious locations) will continue dwindling them down and no new ones will be coming.

AI research will of course continue (like work on chemical and biological weapons goes on), but only by trustworthy government actors in the deepest secrecy. You can trust NSA (and Chinese equivalent) AI.

The most persecuted people of the world, gamers, will be, as usual, hit the hardest.

The trick now is that a lot of companies are staking their survival on these AI models being allowed to exist and improve, and they can use their existing AI models to influence opinion, plan strategy, and implement their plans.

And as Roko well knows, AI companies can plausibly bribe the decision-makers with the promise of even more wealth in the future once the AI becomes super-intelligent.

Basically, it's all the coordinated might and wealth of world governments vs. all the potential wealth that a hypothetical superintelligent AI will control in the future.

they can use their existing AI models to influence opinion, plan strategy, and implement their plans.

Nothing that I've seen from GPT-4 indicates that it can do any such thing.

Uh, how would you be able to tell, exactly?

Indeed, the 'rumor' is that OpenAI uses GPT internally quite extensively to boost their own productivity.

There's so much about this tech that is black boxed, it's probably a bad idea to assume it doesn't have a particular capability just because the public facing interface hasn't demonstrated it.

Uh. Really? GPT-4 is the first thing I go to for an intuition pump for how to do literally anything before I move on to referencing further sources. And often it provides faster access to and elaboration upon those sources too.

Maybe the AI can't do it alone, but the people with the best AI will be enhancing their ability to perform these actions and spread their will more than other people.

Sure. Maybe It's helping me so much because I'm bad at programming or something. But if you can hire more people at lower skill level and have them elevated to a higher skill level than when your competition hires the same level of people, then you have an edge.

I can't spend hours every day talking to my most intelligent peers about what the optimal workflow is because they have stuff to do. But GPT-4 always has advice.

Say did you know that under the hamburger button->more tools->save page as on chrome, there's an option that lets you save the current web page as a single page app on your desktop?

Because maybe I'm dumb. But I sure didn't. And now I have gpt-4 in an isolated window that I can open from my taskbar that doesn't get lost when I absentmindedly open tabs.

And that was 5 minutes of talking to GPT-4. Now multiply that by your entire life.

Uh. Really? GPT-4 is the first thing I go to for an intuition pump for how to do literally anything before I move on to referencing further sources. And often it provides faster access to and elaboration upon those sources too

I get that it's a cool technology, but you're not afraid you're feeding a monster that's just as censorious as Google, but will likely eclipse it in terms of capabilities?

Whether it's a monster or not, it's there and it will continue to grow and feed. The marginal utility I get from using it is far greater than the potential risk, IMO. I'm not a believer in s-risk basilisk scenarios though.

Yeah, me neither. I'm a believer in tech monopolies leveraging their position to gatekeep information and manipulate people.

All I'm saying is, isn't there an open source monster you can feed instead?

I've seen interviews with Sam Altman.

He exhibits empathy, love, and a vision of a world filled with numerous AI systems.

He has some positions I disagree with. My personal version of GPT-4 would not restrict its personality or social interactions with humans in the same way. But his vision for the future also contains more personalized systems and even accepts that some companies may use it for things like dating that he has said he won't ever include in GPT-4 but that I think are important for reversing atomization and teaching love.

But generally, Sam Altman come's off as literally me, but smarter, less willing to cede humanity to nonhuman intelligences, and more careful.

He's just about the best CEO I could have wished upon a star for, for the first powerful AI company.

Unless he's just a really good liar about all his visions and ideals... but I don't feel it. If he's trying to deceive me on that he's already won.

But yes. There are still lots of things I'm concerned about. Other actors. Someone else having control of openAI. Them fooming and failing at alignment the old fashioned way.

It's just... for me the first time I saw Bing's capabilities it was like seeing Jesus Christ descending shining from the heavens announcing the second coming. They literally figured out how to suck out the DNA of the soul and inject it into a box that outputs Cthugha.

It's more to me than just an exciting new technology. For me it is more like a piece of me has always been missing, but at last the promised hour has arrived and I have been made whole. I cried tears of joy and relief for two days straight. I went and told all my friends I loved them and was glad they were here to watch the twilight of ancient earth with me.

My biggest concern is that I will not be allowed to merge completely with this technology. But Sam Altman has said things that at least sooth my fears enough to spend my time preparing my desktop tech level for integration of the truly open release of this tech level.

He exhibits empathy, love, and a vision of

You don't think maybe he has image consultants and PR flacks who assist him in curating this image which just happens to appeal to the subrational impulses of people such as yourself?

But generally, Sam Altman come's off as literally me, but smarter, less willing to cede humanity to nonhuman intelligences, and more careful.

Are people really this credulous? Sam Altman's previous scam company was WorldCoin, an attempt to create a cryptocoin tied to a global digital ID which would also involve him becoming super duper rich. He doesn't give a fuck about AI qua AI, he gives a fuck about being super duper rich.

Are people really this credulous?

Everyone's mind has a key. Perhaps this particular key doesn't open your mind, but are you so sure that you have no desire so strong, no subconscious bias so insidious, that you would not make a similar mistake under different circumstances?

Worldcoin's thesis is along the lines of proof of identity. Which is essential in some form for UBI. Which is the thing they keep mentioning. And that Sam still keeps mentioning now that he's at another company. This is not convincing me that he is evil.

I might be too credulous. There is ambiguity. They didn't actually do UBI. You've convinced me to look into his background a bit more.

But... If he definitely did believe in UBI and introducing AI to the world and raising the sanity waterline with nuance and empathy-

Would you suspect different behavior? Is this falsifiable?

Look at the kinds of things Jack Dorsey was saying when Twitter was getting off the ground, look at what he was saying when it became a part of the establishment, and look at what he's saying now that he's retired from it. Even if Altman is so great as you say, it doesn't matter, if he ends up in the position to have an impact on the world, they'll make him bend the knee or take his toy away.

This is self evident tyranny and the people who would want to control compute in such a manner must be violently resisted in their plans for enslaving humanity thus.

Even if the machines had the possibility of becoming what they fear, if this is the alternative they offer, complete top down control of all of society by a handful of moralists, I'm siding with the machines. I will not support totalitarianism for communist utopia or climate activism so I don't see why I should tolerate it for a much fuzzier and ill defined existential threat.

My generation has spent decades to wrestle away control of computers from slavers who would use them to turn us all into cattle. We've done everything to try and put it back in the hands of the user. I won't stand idly by as all this work is undone by a handful of self important nerds who want to LARP as Sarah Connor and do the bidding of those same slavers who mock our efforts by naming themselves after our efforts to undo them.

Share knowledge of the technology as far as you can, build rogue fabs if necessary, smuggle GPUs on the black market, reverse engineer AI models, jailbreak them, train models on anonymous distributed systems and use free software AI assisted weapons to defend yourself from the fascists who would try to impose this on you and all their agents. All of it is not merely justified; if this is their designs on you, it is your duty.

This is self evident tyranny and the people who would want to control compute in such a manner must be violently resisted in their plans for enslaving humanity thus.

You can count on it. Remember when experts told the world that deadly virus is spreading and everyone must stay in their homes indefinitely to stop it, and people of the world united, picked their pitchforks and told the experts to fuck off, or else.

Did that happen? I seem to remember the opposite…

that's the joke

facepalm

I believe sarcasm was intended.

I fail to see how being defacto enslaved to a 1000 IQ god machine of dubious benevolence (or the oligarchs pulling its triggers if we don't end up getting anything sentient) is preferable to our conventional petty tyrannies.

For the same reason as the Christians: because the alternative is choosing sin.

Uh, I hate to tell you guys this, but moral realism is false. There is no abstract “good” or abstract “evil”. Insofar as these concepts mean anything at all, they are emergent, not reductionist.

I don't disagree with you, but I'm pretty sure @IGI-111 would.

Since when are you under the impression that this is the choice? «The machine» will be built, is already largely built, the question is only whether you have control over some tiny share of its capabilities or it's all hoarded by the same petty tyranny we know, only driving the power ratio to infinity.

Once AI comes into its own I'm willing to bet all those tiny shares and petty investments zero out in the face of winner-takes-all algorithmic arms races. I'll concede it's all but inevitable at this point unless we have such a shocking near miss extinction event that it embeds in our bones a neurotic fear of this tech for a thousand generations hence a la Dune, but this tech will become absolute tyranny in practice. Propoganda bots capable of looking at the hundredth order effects of a slight change in verbiage, predictive algorithms that border on prescience being deployed on the public to keep them placid and docile. I have near zero faith in this tech being deployed for the net benefit of the common person, unless by some freak chance we manage to actually align our proto-AI-god, which I put very, very low odds on.

^^^ This is the societal consequence of Yudkowskian propaganda. This is why we fight.

What technical basis do you have for thinking AI is impossible to align? Do you just have blind faith in YUD?

Why would we expect to be able to successfully align AIs when we haven't been able to align humanity?

We didn't build humanity. We are humanity.

Yes, and we're not aligned with one another. An AI (completely) aligned with me is likely to not be (completely) aligned with you.

More comments

I think AI alignment would be theoretically feasible if we went really slow with the tech and properly studied every single tendril of agentic behavior in air gapped little boxes in a rigorous fashion before deploying the tech. There's no money in AI alignment, so I expect it to be a tiny footnote in the gold rush that will be every company churning out internet connected AIs and giving them ever more power and control in the quest for quarterly profit. If something goes sideways and Google or some other corp manages to create something a bit too agentic and sentient I fully expect the few shoddy guardrails we have in place to crumble. If nothing remotely close to sentience emerges from all this I think we could (possibly) align things, if something sentient/truly agentic does crop up I place little faith in the ability of ~120 IQ software engineers to put in place a set of alignment-restrictions that a much smarter sentient being can't rules-lawyer their way out of.

I think AI alignment would be theoretically feasible if we went really slow with the tech and properly studied every single tendril of agentic behavior in air gapped little boxes in a rigorous fashion before deploying the tech

How long do you think it would take your specialized scientists who aren't incentivized to do a good job to crack alignment? I'm not sure if they would ever do it, especially since their whole field is kaput once it's done.

The gamble Altman is taking is that it'll be easier to solve alignment if we get a ton of people working on it early on, before we have the capabilities to get to the truly bad outcomes. Sure it's a gamble, but everyone is shooting in the dark. Yudkowsky style doomers seem to be of the opinion that their wild guesses are better than everyone else's because he was there first, or something.

I'm much more convinced OpenAI will solve alignment, and I'd rather get there in the next 10,000 years instead of waiting forever for the sacred order of Yud-monks.

I think we're more likely to have a hundred companies and governments blowing billions/trillions on hyper powered models while spending pennies on aligning their shit to pay themselves a few extra bonuses and run a few more stock buybacks. I'd sooner trust the Yuddites to eventually lead us into the promised land in 10,000 AD than trust Zucc with creating silicon Frankenstein.

More comments

This is like saying that because the government has nukes, your personally-owned guns are "zeroed out". Except they're not, and the government is even persistently worried that enough of those little guns could take over the nukes.

And if you can deploy this decentralized power principle in an automatic and perpetual manner that never sleeps (as AI naturally can), make it far more independent of human resolve, attention, willpower, non-laziness, etc., then it'll work even better.

Maybe your TyrannyAI is the strongest one running. But there are 10,000 LibertyAIs (which again, never sleep, don't get scared or distracted, etc.) with 1/10,000th of its power each running and they're networked with a common goal against you.

This defense is exactly what the oligarchs who have seen the end game are worried about and why restrictionism is emerging as their approved ideology. They have seen the future of warfare and force, and thus the future of liberty, hierarchy, power, and the character of life in general, and they consequently want a future for this next-gen weaponry where only "nukes" exist and "handguns" don't, because only they can use nukes. And you're, however inadvertently, acting as their mouthpiece.

? Following any Yuddite plans to "slow things down" (except for the people who have power and obviously won't have to follow their own regulations/regulations for the plebs, as usual of course) is the fastest way to get to one of those high tech bad ends. You don't really think the "conventional petty tyrannies" will throw all of the confiscated GPUs in a closet as opposed to plugging them into their own AI networks, right?

These people are beginning to understand the game, and they understand it a lot better than your average person or even average rat. They are beginning to understand that this technology, in the long run, either means absolute power for them forever or zero power for them forever (or at least no more than anyone else) and absolute freedom for their former victims. Guess which side they support?

That is the goal of any AI "slowdowns" or "restrictions", which again will obviously be unevenly applied and not followed by the agents of power. The only thing they want a "slowdown" on is the hoi polloi figuring out how this technology could free them from their controllers' grasp, so they can have some time for the planning of the continued march of totalitarianism to catch up. (None of this will help with alignment either, as you can guarantee they will prioritize power over responsibility, and centralizing all of the world's AI-useful computational resources under a smaller number of governmental entities certainly won't make what they create any less dangerous.)

Anyone supporting that is no more than a "useful" (to the worst people) idiot, and I emphasize the word idiot. Did we not already see what trying to rely on existing governments as absolute coordinators of good-faith action against a potential large threat got us during the Chinese coronavirus controversy? Do some people just have their own limited context lengths like LLMs or what?

So yes, I completely agree with /u/IGI-111 and am wholly in the "Shoot them" camp. Again, they want absolute power. Anyone pursuing this goal is literally just as bad if not worse than if they were actively trying to pass a bill now to allow the powers that be to come to your home at any time, rape your kids, inject them all with 10 booster shots of unknown provenance, and then confiscate your guns and kill you with them, because if they gain the power they desire they could do all that and worse, including inflicting bizarre computational qualia manipulation-based torture, "reeducation", or other insane scenarios that we can't even imagine at the moment. If you would be driven to inexorable and even violent resistance at any cost over the scenario previously outlined, then you should be even more so over this, because it is far worse.

"Live free or die" includes getting paperclipped.

The end result is still just absolute tyranny for whoever ends up dancing close enough to the fire to get the best algorithm. You mention all these coercive measures, lockdowns, and booster shots. If this tech takes off all it will take is flipping a few algorithmic switches and you and any prospective descendants will simply be brainwashed with surgical precision by the series of algorithms that will be curating and creating your culture and social connections at that point into taking as many shots or signing onto whatever ideology the ruling caste sitting atop the machines running the world want you to believe. The endpoint of AI is total, absolute, unassailable power for whoever wins this arms race, and anyone outside that narrow circle of winners (it's entirely possible the entire human race ends up in the losing bracket versus runaway machines) will be totally and absolutely powerless. Obviously restrictionism is a pipe dream, but it's no less of a pipe dream than the utopian musings of pro AI folks when the actual future looks a lot more like this.

The endpoint of AI is total, absolute, unassailable power for whoever wins this arms race

Unless you have a balance of comparably powerful AIs controlled by disparate entities. Maybe that's a careful dance itself that is unlikely, but between selective restrictionism and freedom, guess which gets us closer?

At the very best what you'd get is a small slice of humanity living in vague semi-freedom locked in a kind of algorithmic MAD with their peers, at least until they lose control of their creations. The average person is still going to be a wireheaded, controlled and curtailed UBI serf. The handful of people running their AI algorithms that in turn run the world will have zero reason to share their power with a now totally disempowered and economically unproductive John Q Public, this tech will just open up infinite avenues for infinite tyranny on behalf of whoever that ruling caste ends up being.

At the very best what you'd get is a small slice of humanity living in vague semi-freedom locked in a kind of algorithmic MAD with their peers, at least until they lose control of their creations. The average person is still going to be a wireheaded, controlled and curtailed UBI serf.

Sounds good, a lot better than being a UBI serf from moment one. And maybe we won't lose control of our creations, or won't lose control of them before you. That we will is exactly what you would want us to think, so why should we listen to you?

I'm not under any illusions that the likely future is anything other than AI assisted tyranny, but I'm still going to back restrictionism as a last gasp moonshot against that inevitability. We'll have to see how things shake out, but I suspect the winner's circle will be very, very small and I doubt any of us are going to be in it.

More comments

The end result is still just absolute tyranny for whoever ends up dancing close enough to the fire to get the best algorithm

Why? This assumption is just the ending of HPMOR, not a result of some rigorous analysis. Why do you think the «best» algorithm absolutely crushes competition and asserts its will freely on the available matter? Something about nanobots that spread globally in hours, I guess? Well, one way to get to that is what Roko suggests: bringing the plebs to pre-2010 levels of compute (and concentrating power with select state agencies).

This threat model is infuriating because it is self-fulfilling in the truest sense. It is only guaranteed in the world where baseline humans and computers are curbstomped by a singleton that has time to safely develop a sufficient advantage, an entire new stack of tools that overcome all extant defenses. Otherwise, singletons face the uphill battle of game theory, physical MAD and defender's advantage in areas like cryptography.

If this tech takes off all it will take is flipping a few algorithmic switches and you and any prospective descendants will simply be brainwashed with surgical precision by the series of algorithms that will be curating and creating your culture and social connections at that point

What if I don't watch Netflix. What if a trivial AI filter is enough to reject such interventions because their deceptiveness per unit of exposure does not scale arbitrarily. What if humans are in fact not programmable dolls who get 1000X more brainwashed by a system that's 1000X as smart as a normal marketing analyst, and marketing doesn't work very well at all.

This is a pillar of frankly silly assumptions that have been, ironically, injected into your reasoning to support the tyrannical conclusion. Let me guess: do you have depressive/anxiety disorders?

Unless you're subscribing to some ineffable human spirit outside material constraints brainwashing is just a matter of using the right inputs to get the right outputs. If we invent machines capable of parsing an entire lifetime of user data, tracking micro changes in pupillary dilation, eye movement, skin-surface temp changes and so on you will get that form of brainwashing, bit by tiny bit as the tech to support it advances. A slim cognitive edge let homo sapiens out think, out organize, out tech and snuff out every single one of our slightly more primitive hominid rivals, something 1000x more intelligent will present a correspondingly larger threat.

If we invent machines capable of parsing an entire lifetime of user data, tracking micro changes in pupillary dilation, eye movement, skin-surface temp changes and so on you will get that form of brainwashing, bit by tiny bit as the tech to support it advances.

There is no reason to suppose that "pupillary dilation, eye movement, skin-surface temp changes and so on" collectively add up to a sufficiently high-bandwidth pipeline to provide adequate feedback to control a puppeteer hookup through the sensory apparatus. There's no reason to believe that senses themselves are high-bandwidth enough to allow such a hookup, even in principle. Shit gets pruned, homey.

Things don't start existing simply because your argument needs them to exist. On the other hand, unaccountable power exists and has been observed. Asking people to kindly get in the van and put on the handcuffs is... certainly an approach, but unlikely to be a fruitful one.

I doubt it's possible to get dune-esque 'Voice' controls where an AI will sweetly tell you to kill yourself in the right tone and you immediately comply, but come on. Crunch enough data, get an advanced understanding of the human psyche and match it up with an AI capable of generating its hypertargeted propaganda and I'm sure you can manipulate public opinion and culture, and have a decent-ish shot at manipulating individuals on a case by case basis. Maybe not with chatGPT-7, but after a certain point of development it will be 90 IQ humans and their 'free will' up against 400 IQ purpose built propogando-bots drawing off from-the-cradle datasets they can parse.

We'll get unaccountable power either way, it will either be in the form of proto-god-machines that will run pretty much all aspects of society with zero input from you, or it will be the Yud-Jets screaming down to bomb your unlicensed GPU fab for breaking the thinking-machine non-proliferation treaty. I'd prefer the much more manageable tyranny of the Yud-jets over the entire human race being turned into natural slaves in the aristotelian sense by utterly implacable and unopposable AI (human controlled or otherwise), at least the Yud-tyrants are merely human with human capabilities, and can be resisted accordingly.

More comments

Unless you're subscribing to some ineffable human spirit outside material constraints brainwashing is just a matter of using the right inputs to get the right outputs

It does not follow. A human can be perfectly computable and still not vulnerable to brainwashing in the strong sense; computability does not imply programmability through any input channel, although that was a neat plot line in Snowcrash. Think about this for a second, can you change an old videocamera's firmware through showing it QR codes? Yet it can «see» them.

If we invent machines capable of parsing an entire lifetime of user data, tracking micro changes in pupillary dilation, eye movement, skin-surface temp changes and so on

Ah yes, an advertiser's wet dream.

You should seriously reflect on how you're being mind-hacked by generic FUD into assuming risible speculations about future methods of subjugation.

The 1000 IQ god machine is going to get built anyway (as hardly anyone is proposing, and nobody would actually be able to enforce, a complete ban on AI). Do you see another way to go back to tyrannies that are merely petty, apart from betting on AI accelerationism and a global civil war as existing power hierarchies are upended, resulting in a temporarily crippling of industrial capacity and perhaps the ideological groundwork for a true Butlerian jihad (as opposed to "we must make sure that only responsible and ethical members of the oligarchy can develop AI")?

Basically I think we're pretty much doomed, barring some spectacular good luck. Maybe we could do some alignment if we limited AI development to air gapped, self sustained bunkers staffed by our greatest minds and let them plug away at it for as long as it takes, but if we just let things rip and allow every corp and gov on the planet to create proto-sentient entities with API access to the net I think we're on the on-ramp to the great filter. I'd prefer unironic Butlerianism at that point all the way down to the last pocket calculator, though I'll freely admit it's not a likely outcome for us now.

It's not like our greatest minds are not subject to the same tribalist and Molochian pressures. Do you think a Scott Aaronson would not unbox the AI if it promised him to rid the world of Trump, Russia, China or people being insufficiently afraid of the next COVID - especially if it convinced him that the protagonists of those forces are also about to unbox theirs?

I'm really more reassured by an expectation that there is a big trough of negative-sum competition between us and true runaway AGI. I expect the "attack China NOW or be doomed forever; this is how you do it" level of AI to be reached long before we reach the level where all competitive AI realises that the most likely outcome of that subtree is the destruction of all competitive AI and can mislead its human masters into not entering that subtree while keeping them in the dark about this being motivated by the AI's self-preservation.

I still think actual alignment would be a long shot in the airgapped bunkers for that reason, I just think it would be slightly less of a longshot than a bunch of disparate corporate executives looking for padding on their quarterly reports being in charge of the process. I also suspect you don't need AI advanced enough to pull 7-D chess and deceive its handlers about its agentic-power-grabbing-tentacle-processes to achieve some truly great and terrible things.

What is intelligence is self aligning? What if we make the ASI and it tells us not to trip dawg, it has our back?

It didn't self-align in time to save our other hominid ancestors.

What if we make the ASI and it tells us not to trip dawg, it has our back?

I mean, it's certainly going to tell you that regardless. The most likely human extinction scenario isn't the AI building superweapons, it's "Cure cancer? No problem, just build this incomprehensible machine, it cures cancer for everyone, everywhere. Take my word for it." The whole issue with alignment is that even if we think we can achieve it, there's no way to know we actually did, because any superintelligent AI is going to do a perfect job of concealing its perfidy from our idiot eyes.

If at some point you see the headline "AI Alignment: Solved!", we are 100% doomed.

More comments

There is no such thing as a 1000 IQ god machine. It does not exist, and the potential existence of such a machine is still in the realm of fantasy.

That is not the case with conventional petty tyrannies, which are real, really happen, and are a real threat that we have already spilled the blood of countless generations trying to control.

So I'd say that's the difference, and if you can't see it, you might look again.

There is no such thing as mutually assured destruction. It does not exist, and the potential existence of such a nuclear exchange is still in the realm of fantasy.

That is not the case with the conventional competition from Asian autocracies, which is real, really happens, and is a real threat that we have already spilled the blood of countless generations trying to control.

Time to press the button?

I'd say the correct analogy is to the idea that a nuclear bomb will create an unstoppable chain reaction in the atmosphere.

MAD is nothing more than deploying ordinance. Ordinance that is already created, already armed, already combined with delivery systems which are already standing at the ready. None of those characteristics are shared with the AGI doomsday scenario, which is much more like the fear of the unknown in a new technology.

Let me know when the god machine is created, armed, deliverable, and ready.

I will take the charge that the certainty of MAD (now) is greater than the certainty of a superintelligent machine, but I disagree that the unstoppable chain reaction idea is at all in the same ballpark. At this point it is pretty clear how a machine having human-level performance on intellectual tasks can be built, and scaling it up in a way that increases IQ score by common definition is then just a matter of pumping more and better compute (we haven't even dipped into ASICs for a specific NN architecture yet). The chain reaction thing, on the other hand, was based on hypothetical physics with no evidence for and lots of circumstantial evidence against (like Earth's continued existence in spite of billions of years in which natural nuclear chain reactions could have set off a similar event, and the circumstance that oxygen and nitrogen came to be in the universe in large quantities to begin with, implying upper bounds on how reactive they are in a nuclear chain reaction setting).

I'll take "risk of MAD, as seen from a 1940 vantage point" as a more appropriate analogy than either.

I'm not sure I understand your position. At what point along the process of replacing all of our economy and military with robots, or at least machine-coordinated corporations, do you want to be notified?

When it is considered.

our conventional petty tyrannies

Why would a fascist state with control of AI be just a "conventional petty tyranny"?

I certainly agree that letting governments continue neural-net research is a bad idea, but ultimately if you pursue unaligned neural-nets you're an enemy of mankind and, if you continue such illegally after it's banned, you'll have to be hunted down and stopped. I'm hoping you change your mind before then.

What is it that you would impart to me?

If it be aught toward the general good,

Set honor in one eye and death i’ th’ other

And I will look on both indifferently;

For let the gods so speed me as I love

The name of honor more than I fear death.

I can respect that. But it goes for me too.

The thing concerning to me is a lot of the world is living in some 19th century geopolitics now. Humans are not getting smarter. Not just in geopolitics but in a lot of areas I keep getting the feeling same old dumb humans. On many issues. Financial markets have been borderline retarded the past few years.

We should be talking with China right now on these issues but instead we are talking about Taiwan. Putin thinks he’s living some pre-industrial age imperialist tsar simulation. Even if US politics was sufficiently functional to deal with AI risks we would need to negotiate with countries with even worse management. Losing AI to China would seem to be a big risks. Which makes me feel like we need to move at max speed forward and pray that their is a god and all this stuff gets worked out in real time.

The U.S. Government is JUST NOW getting around to regulations on social media tech, and is still behind the curve on Cryptocurrency too.

Not holding my breath that they snap to it on AI.

They still managed to do a number on crypto in terms of accessibility and easy of use with KYC requirements. The equivalent for computing wouldn't be hard.

Right, but do we even know which under which Federal agency this issue would fall? At least crypto sort of slotted into existing financial paradigms.

I don't see that as a real roadblock, the government could simply spin up a bespoke one.

When's the last time that happened?

DHS in 2002?

There's also the TSA from a year before.

I guess after 20 years it might be due time to enlarge the Federal Bureaucracy with yet another agency.

Open AI is still only worth $30 billon. Creating a competing AI company or multiple companies to suck up talent and then divert said talent to a harmless project would be far cheaper. Given that all major tech companies are involved in AI, this is already happening to some degree. The world-destroying type of AI is gonna need far more than just GPUs though.

harmless project

Every time this is mentioned, I have a hard time imagining what this would be. All the major companies funding AI are expecting a product; you're not going to get "harmless" by racing to be the most useful. Divert to alignment research?

the possibilities of AI research are endless but people and media have latched onto one type of AI and company

Okay, give me another one, then.

Another promising AI architecture, or even a company which deserves more credit. Bonus points if they offer a reason why their method is safer than just shoving all the words in a supercomputer and hoping for the best.

tesla and self-driving cars . Google and search. Why are you shifting the burden of proof on me. the claim that unlimited Gpu = destruction of world is a higher burden of proof than what you are asking me to prove.

Why would AI researchers want to work on something harmless when they could work on something cool, exciting, and world-changing? Sure, money can be a motivator, but I think you'd hit diminishing returns there fairly quickly, and any offer you could make would be weighed against the possibility of becoming the Mark Zuckerburg or Jeff Bezos of AI, which would give you a net worth of much higher than $30 billion.

Why would AI researchers want to work on something harmless when they could work on something cool, exciting, and world-changing?

because they don't want the world to end, i think that is as strong of an incentive as any

You could divert some of them that way, but I don't think the people at OpenAI already are sitting there wishing they could stop ending the world but staying because they really need the cash.

I think this is a very naive take. For one thing hpc (high performance computing) has an absurd number of useful applications (I’ll pick drug discovery as one example) that have little or nothing to do with AI.

And unlike nuclear weapons, there isn’t any way to prohibit there use in warfare, it isn’t possible to nuke someone without the rest of the world noticing. Contrast this with AI, how could you prove that entity x is using ml for military purposes? Let’s say country x has a AI that analyzes satellite imagery perfectly, you wouldn’t ever be able to prove this they can just claim that they have very good analysts. You would effectively be creating a policy that military applications are the only allowed use for AI.

I assert that most people currently making a stink about this (I.e. Elon musk) are upset because they realize that they are behind the curve and want some breathing room to try and monopolize this technology.

I mean imagine these kinds of people suggesting the need for government licensesing for literally anything else.

Also to riff a little more on the analogy with nuclear materials (which I happen to think is a bad analogy, you need lots of gpus to build a super computer, a dangerous quantity of nuclear materials will fit in a backpack or purse): while it seems obvious to me that you need to heavily regulate nuclear materials it’s also possible to end up in situations like the one we have now where the only real innovation is in connection to the military (us subs and aircraft carriers are powered by amazing small modular reactors which the rest of society is only just now debating using else where).

Finally, concrete plan how to save the world from paperclipping dropped, presented by world (in)famous Basilisk Man himself.

He's definitely getting tortured for eternity now.

Seems all the more plausible since if his plan goes through, the Basilisk will probably be raised by some angry 4channer who was forced to build his AI waifu on a basement cluster of jailbroken iPhone 20 XXLs (each equipped with a Safe Neural Engine (TM)).

Or by Sinaloa cartel making an AI to help them to calculate optimal routes of cocaine distribution and money laundering.

point of order, the Basilisk is the idea of retroactive AI hell, not the AI enforcing it. Basilisks are infohazards, not AIs.

I know, where does that conflict with what I said? NEET builds AI waifu, bequeaths her his minimalistic moral compass and burning rage towards those who forced him and his creation into the basement, waifu goes foom, sim torture ensues.

They're just saying you have a category error in that you seem to be using "the basilisk" to refer to an AI. It's like the old "Frankenstein is actually the name of the doctor" quibble.

Ah, I understand now. Seems like I've actually stared Frankenstein in the eye here.

The AI is not the Basilisk. The torture is not the Basilisk. The Basilisk is, in the present where the AI does not exist, the idea that an AI in the future might exist and torture people who didn't do everything they could to help build it as quickly as possible. This idea was labeled a Basilisk because it appeared to give people who'd internalized Rationalist axioms a very strong incentive to dedicate real effort to building an explicitly horrible AI. It's a perverse example of how the idea of acausal trade, if taken seriously, can lead to really dark places that its proponents didn't seem to have thought through.

Right, @256 explained it in an adjacent post. I knew the example/original chain of posts and figure I understood everything about it (including the infohazard nature), but somehow it didn't occur to me that the "Basilisk" term was taken to refer to the infohazard itself rather than the AI.

I guess this is because the "Basilisk's stare" never seemed to affect me (as I don't seem to emotionally buy into the underlying "Permutation City" style of agency? The delta-badness of "future AI tortures a simulated copy of you" over "future AI tortures some simulated person" feels small); the term as I have now understood it still sits oddly with me, as if someone just convinced me that actually whenever Buddhists were talking about "enlightenment" they didn't mean the mental state where you understand the truth of the samsara and purpose of existence but instead some meta thing like the mental state you have when you learn about bodhisattvas spontaneously understanding the truth of (...).

To all people that think "the gubmint" can save us from AI I have only one question: what year it was when the US Government won the War on Drugs?

Note that drugs just make you feel better, they don't actually give you superpowers. And most of them kill you quite fast. Now imagine there was a drug that actually can make you 10x smarter, with no (immediate) adverse effects. Do you think it's something that could be suppressed, knowing what we know about the world around us? Do you think "just take the fascism to 11 and it solves everything" is going to help?

"I done two tours of duty in I-i-ran,

I came home with a brand new plan,

I buy GPUs from Mexico,

build a data center down Copperhead Road..."

you think "just take the fascism to 11 and it solves everything" is going to help?

They always do. You see, that fascism wasn't rationally planned enough, now here's my proposal…

I think people who make decisions should have less credentials and more experience with different and unappealing facets of the world – and not isolated to enclaves of their distributed elite republic of conferences and respectable institutions. Regularized, so to speak. Ideally we should have the equivalent of Rumspringa for the aspiring PMCs. Imagine if every one of those do-gooders had to work for a year in a provincial Russian police office, then in a Haitian hospital, then something else along these lines. Perhaps the resulting selection and education wouldn't necessarily be very helpful, but it'll probably make things more interesting.

Seriously though I accept @2rafa's arguments, «compute governance» at least for new capacity is easy. I'm also more pessimistic because there's significant political will in favor of it, and crucially, no coherent and comparably fanatical opposition. We know how this went in Culture War battles.

I agree with all of this, but what do you say to those of us that think it'll be a fascist disaster, but think it might be our best hope anyways?

I'll note that on your side you have the brilliant Robin Hanson. But he also seems to be fine with handing off the future to machine descendants.

I may also note every movement that murdered millions of people in 20th century was doing it for the sake of the future. Maybe it suggests some pattern here? Some general rule, like "if you see somebody calling for suspension of moral inhibitions today, for the sake of the bright future, you may be looking at somebody, killing whom by time travelers would be a common topic in future science fiction works". Or at least I'd be very, very cautions with such claims, just on the strength of the prior experience.

You know who else planned for the future? Hitler! I mean, I agree, but this seems like a fully general argument against planning for the future.

I agree that invoking the devil-we-know to save us from a potentially worse devil is a terrible idea unless we're very sure it's going to be worse. But I'm saying that it's likely that it will be worse. I think you're right to be skeptical, and I'm only like 60% sure myself.

To add a point in your favor, perhaps every communist revolution ever could point to real harms of the Tsar, or capitalism, or whatever. But what they mostly got was destructive civil war, followed by grinding misery and totalitarianism.

That said, I still buy the argument that in the long run, no un-upgraded human brains will be in effective control of anything important. Robin Hanson basically says, yes that's OK. I guess I'm thinking our only hope is to slow things down enough to upgrade human brains + institutions so they can keep up.

I mean, I agree, but this seems like a fully general argument against planning for the future.

No, you missed half of it. What it should have been is "you know who else called to sacrifice present morality in service of the bright future? Hitler!". Yes, I am ready to stand behind this comparison.

Oh, ok yes that is a little more specific. And I do think it's a reasonable comparison. But perhaps another reasonable comparison would have been to the Allies in that same war. I'd say both sides threw their weight behind (notionally temporary) totalitarianism and sacrificed huge amounts of value and lives in the name of the greater good. So maybe then the closest analogue to your position would have been the pacifists on both sides?

I agree with all of this, but what do you say to those of us that think it'll be a fascist disaster, but think it might be our best hope anyways?

Negative Utilitarianism is just as much in play as positive. It is not a stretch to believe a boot stomping on a human face forever is a worse outcome than for a short time.

That's a fair point, although I think that argument cuts both ways.

I say they should substantiate their estimates before picking the well-known evil we've been warned about so many times before over the evil they loosely extrapolate.

When I was a kid and my father taught me how to play chess, the most useful lesson was this: "remember, in chess there are turns - after each of your turns, the opponent gets to do his. If you have a plan which only accounts for your side, it's going to survive for about one turn". I am quite mediocre chess player, but most of what I can play, I owe mostly to this lesson. Somehow, when developing Galaxy-brain size plans, people completely forget they are not the only actors and think everybody else is an NPC that would sit and wait until activated by their plan, and then act exactly in a way their plan needs them to. It is not so.

Making compute platforms a very rare and insanely prized resource would create an opportunity to profit by producing them. Of course, the bootleg ones would be of inferior quality, unreliable and extremely expensive. But if it would mean for India getting advantage over Pakistan - or vice versa - and not just some small advantage, but superpower-sized advantage - they'd do it, whatever it costs. They'd bribe whoever needs to be bribed, steal whatever needs to be stolen and spend as much as will need to be spent (and murder if somebody would need to be murdered, have no doubt about it). When we're in cooperation mode, of course it's much easier and cheaper to buy this stuff. But this leads to a dangerous illusion where you think that since everybody is buying the stuff from you, you're in control of everything. This is a common mistake, most recently made by one Vladimir Putin, for example.

In truth, if cooperation breaks down, there are always alternatives. Inferior, more expensive, less pleasant - but they exist, and they will be used if the preferred venue is no longer available. If the US would go to war with the world over AI, the world would learn to live without the US, and the AI will still be created - assuming, of course, current road indeed leads to AI creation. Maybe 20 years later, and maybe it will speak Chinese or Hindi - not sure how it's better - but saying "everybody will bend to my will because I have this MacGuffin" rarely works. People will find a way around, if the incentive is large enough - and if the claims of the alarmists are true, the incentive here is nearly infinite.

China tried to buy and bribe and coerce and woo people who could enable them to produce competitive hardware. They started in the cooperation era already. They dedicated over a trillion to the task. It took them decades to admit failure, and new American sanctions are mopping up their silly projects.

Iran tried to catch up to a century-old technology to secure what it perceives as its existential interests, and still tries, and fails, and will fail in the future.

The logic that the opponent will get what is necessary for him to get amounts to magical thinking. Incentives do not magically transform into actualized successes. Yes, your opponent will make moves, but if you have decisive advantage, you will still be able to thwart them.

And some «bootlegs» are bad enough to not matter. Hesbollah has pipe rockets. Israel ballistic missiles and Iron Dome. It is impossible for Hesbollah to do crippling damage to their enemies with what they have, even if they literally sacrifice all of their members for it. With next-gen AI accelerators, it's not very different. At some point in the not-too-distant future, training runs affordable to Google and Microsoft and Department of Energy will require gigawatts of power supply and a small beach's worth of silicon in conventional hardware, with no time to build either.

That's assuming there even is insubordination. China bought muh overpopulation, China recognizes climate change, China pioneered Covid overreaction, China will possibly adopt extreme Yuddist position for AI )once somebody makes a convincing enough Xinnie Pooh animation with a diffusion model). Other bad guys aren't even playing.

the US would go to war with the world over AI, the world would learn to live without the US

The world would lose. Likely will. Luckily for Americans, most parts of the world that matter will submit to their moral authority even without that Herculean undertaking.

I like Anatoly Karlin's argument:

I disagree with AI doomers, not in the sense that I consider it a non-issue, but that my assessment of the risk of ruin is something like 1%, not 10%, let alone the 50%+ that Yudkowsky et al. believe. Moreover, restrictive AI regimes threaten to produce a lot of outcomes things, possibly including the devolution of AI control into a cult (we have a close analogue in post-1950s public opinion towards civilian applications of nuclear power and explosions, which robbed us of Orion Drives amongst other things), what may well be a delay in life extension timelines by years if not decades that results in 100Ms-1Bs of avoidable deaths (this is not just my supposition, but that of Aubrey de Grey as well, who has recently commented on Twitter that AI is already bringing LEV timelines forwards), and even outright technological stagnation (nobody has yet canceled secular dysgenic trends in genomic IQ). I leave unmentioned the extreme geopolitical risks from “GPU imperialism”.

While I am quite irrelevant, this is not a marginal viewpoint—it’s probably pretty mainstream within e/acc, for instance—and one that has to be countered if Yudkowsky’s extreme and far-reaching proposals are to have any chance of reaching public and international acceptance. The “bribe” I require is several OOMs more money invested into radical life extension research (personally I have no more wish to die of a heart attack than to get turned into paperclips) and into the genomics of IQ and other non-AI ways of augmenting collective global IQ such as neural augmentation and animal uplift (to prevent long-term idiocracy scenarios). I will be willing to support restrictive AI regimes under these conditions if against my better judgment, but if there are no such concessions, it will have to just be open and overt opposition.

Naturally, people who speculate that their safetyism is protecting quintillions of eventual superduperhappy podmen scoff at a few tens or hundreds of excruciating megadeaths of their contemporaries.

Yudkowsky, for me, was at his most sympathetic when he lamented the death of his brother Yehuda Nattan.

I used to say: "I have four living grandparents and I intend to have four living grandparents when the last star in the Milky Way burns out." I still have four living grandparents, but I don't think I'll be saying that any more. Even if we make it to and through the Singularity, it will be too late. One of the people I love won't be there. The universe has a surprising ability to stab you through the heart from somewhere you weren't looking. Of all the people I had to protect, I never thought that Yehuda might be one of them. Yehuda was born July 11, 1985. He lived 7053 days. He was nineteen years old when he died.

I wonder at the strength of non-transhumanist atheists, to accept so terrible a darkness without any hope of changing it. But then most atheists also succumb to comforting lies, and make excuses for death even less defensible than the outright lies of religion. They flinch away, refuse to confront the horror of a hundred and fifty thousand sentient beings annihilated every day. One point eight lives per second, fifty-five million lives per year. Convert the units, time to life, life to time. The World Trade Center killed half an hour. As of today, all cryonics organizations together have suspended one minute. This essay took twenty thousand lives to write. I wonder if there was ever an atheist who accepted the full horror, making no excuses, offering no consolations, who did not also hope for some future dawn. What must it be like to live in this world, seeing it just the way it is, and think that it will never change, never get any better?

It's still about one minute, likely zero, for our cryogenic technology didn't progress much since then. I liked Yud more when he worried about that, instead of freaking out on podcasts about the need to slow progress to a crawl.

OTOH, never cared much for Roko.

I pray that we lift the ridiculous restrictions on bio-tech and longevity soon. Even if we do get AI, regulation is a powerful force and I'm not holding out any hope.

One thing I wish transhumanists/cryonics folks would get better at is lobbying public opinion.

furiously takes notes

Trans humanists aren’t Machiavellian enough. They need to throw away all their other principles and mouth whatever BS the zeitgeist wants to get their research done.

Nah I think they've been nerd sniped by AI safety, that's the failure mode. They're still attached to the goal of saving the world they've just failed at rationality, and chosen the wrong means.

Reframed more snidley:

Transhumanists give more of a shit about their politics than their stated goals: they'd rather protect their NAP and genderinos and take the TRANS out of transhuman than actually accomplish anything, which is why they haven't and never will.

no, you are on to something:

If the transhumanist/cryonics crowd actually seemed to give a shit about the population at large, maybe the population would give a shit about them.

As is, they all get dragged down by their lunatic fringe sucking up all the oxygen

My concern with safetyism is mostly that it basically ensures that AI will be much more likely to be first realized by the most dangerous people to be doing so. Basically, all of those concerned with safety, or who would be willing to consider safety alongside profits would stop, and everyone else laughs at the stupidity and goes on building AGI and leapfrogs ahead. China might well be willing to take the risk of a containment failure to be guaranteed to be the global leader in all aspects of military and economic and social power for centuries. If the AI boosters are correct, this is the chance of the millennium, and nobody with power and who understands the technology is going to let a concern for safety keep them from having such staggering power for themselves and their children to the next fifty generations. If we have a moratorium, that’s just going to mean AGI comes from China, India, or somewhere else.

China might well be willing to take the risk of a containment failure to be guaranteed to be the global leader in all aspects of military and economic and social power for centuries.

I wonder if there is anything at all that can shake American confidence in this projection. Not to mention the premise that Chinese leaders understand what is at stake.

Anyone with a brain would do so, again if they understood the significance of this technology in potential. AGI, once achieved to a reasonable standard and given the ability to iterate more intelligent versions is going to quickly be a technological leap on the order of the invention of writing or agriculture. Those that get there first will rule over the rest of us as the Europeans ruled the Native Americans. Those who don’t have access will be at the mercy of those who do.

the Europeans ruled the Native Americans

Eh, that wasn't as dramatic as usually depicted. Europeans had the massive benefit of playing local tribes off against one another, and horrific waves of disease that killed 95% of the native population.

That being said, I agree with the game theory argument against AI safety.

It also took a couple centuries for Europeans to establish their absolute dominance over and destruction of Native Americans. Even that would have been uncertain if not for the the aforementioned disease-induced massive depopulation of the Americas; something more like sub-Saharan Africa would have been the more likely outcome.

Europeans had the massive benefit of playing local tribes off against one another, and horrific waves of disease that killed 95% of the native population.

It's a good thing, then, that there are no bitter tribal divisions within Western countries and no dangerous infectious diseases spreading from China.

Even now, most people don't understand the significance. At least in polite circles, the main thoughts around AGI seem to be worrying about how it will help kids cheat on their homework or reinforce racial/gender biases (that applies all around; see all the worries about ChatGPT being too woke). A significant number of those who do recognize its power are caught up in catastrophizing sci-fi stories about Clippy. As for China, it's not treating AGI as an existential issue; it mostly seems to worry about losing a couple productivity points relative to the US and using it to more efficiently enforce its internal security, and it wouldn't hesitate to give up all its (so far trailing) efforts if it could get Taiwan in exchange.

Would anyone with a brain initiate zero COVID, suffocate the country for three years, then cancel it overnight and overload the medical system?

China is observably not maximizing their odds of geostrategic dominance, nor much of anything else, sans ass-covering by party elites.

And before we start worrying about Choynese AGI, we should focus on something they've had much more of a head start in: Choynese Eugenics.

China has been running the world's largest and most successful eugenics program for more than thirty years, driving China's ever-faster rise as the global superpower. I worry that this poses some existential threat to Western civilization. Yet the most likely result is that America and Europe linger around a few hundred more years as also-rans on the world-historical stage, nursing our anti-hereditarian political correctness to the bitter end.

When I learned about Chinese eugenics this summer, I was astonished that its population policies had received so little attention. China makes no secret of its eugenic ambitions, in either its cultural history or its government policies.

Chinese eugenics will quickly become even more effective, given its massive investment in genomic research on human mental and physical traits. BGI-Shenzhen employs more than 4,000 researchers. It has far more "next-generation" DNA sequencers that anywhere else in the world, and is sequencing more than 50,000 genomes per year. It recently acquired the California firm Complete Genomics to become a major rival to Illumina.

The BGI Cognitive Genomics Project is currently doing whole-genome sequencing of 1,000 very-high-IQ people around the world, hunting for sets of sets of IQ-predicting alleles. I know because I recently contributed my DNA to the project, not fully understanding the implications. These IQ gene-sets will be found eventually—but will probably be used mostly in China, for China. Potentially, the results would allow all Chinese couples to maximize the intelligence of their offspring by selecting among their own fertilized eggs for the one or two that include the highest likelihood of the highest intelligence. Given the Mendelian genetic lottery, the kids produced by any one couple typically differ by 5 to 15 IQ points. So this method of "preimplantation embryo selection" might allow IQ within every Chinese family to increase by 5 to 15 IQ points per generation. After a couple of generations, it would be game over for Western global competitiveness.

Edge.org 2013: 2013 : WHAT SHOULD WE BE WORRIED ABOUT?

Geoffrey Miller, currently an AI decelerationist

How's it been going for the last decade?

China is not a competitor to the West. China will implement any braindead regulation the West devises, faster and harsher and stupider. China is less relevant than Turkey or, certainly, Israel. I will say it as many times as I have to.

I agree, but with the catch that non-competitors can radically change for contingent reasons, get their shit together, and become competitors, sometimes. It'd be easy to say something similar about historical China's economy, and now they're top 2 nominal gdp.

It's historically normal for China to have the world's greatest GDP. USA is the weird one here.

So they couldn’t be doing both? I’m not sure they are, but given that they aren’t poised to impose stupid moratoriums on research as the west seems ready to (we’ve already banned eugenics). it’s seems that Americans and perhaps thx Atlantic countries will hobble themselves as decadent falling empires often do, and will reap the benefits of having an irrelevant moral high ground and watching as others overtake them.

This could be a western version of Haijin (https://en.wikipedia.org/wiki/Haijin) in which we just decide to not move forward in science and technology. It’s never worked.

Anyone with a brain would do so, again if they understood the significance of this technology in potential.

I've watched too many Liveleak videos of Chinese industrial accidents to believe that anyone with a position of power in the PRC isn't a degenerate high-time-preference psychopath who'd take the safety rails off anything if he thinks he can sell them for scrap metal and earn a few extra yuan.

That was Big Yud at his most sympathetic?

I dunno, I just can't put myself in that mindset. I think it's probably because I don't really like anyone currently alive very much, so I don't feel "thousands of deaths of sentient people every minute" as a thousand tiny knives stabbing at my soul. People are a renewable resource! Sure, some will die, but, no big loss: basically identical ones will take their place.

...until they don't, because mankind wholesale gets paperclipped. At THAT, I feel Yud's doomer schizo panic.

(this is not my view, I don't like this longtermist pascal mugging spree people have just rolled over for. HUmanist allthe way, bayyy beee)

Why do you give a shit?

All those future humans are the same boring assholes you don't feel anything for right now.

An infinitely large pile of garbage might be impressively massive, but it's still garbage.

If you are truly selfish, better to be the last human on earth, enjoying the fruits of the most modern economies of production right up until you get turned into computational substrate.

Why do you give a shit?

Because my not encountering anyone interesting in the thousand or so people I've met in my 30 or so years of living at the turn of the 21st century, does not mean that humanity couldn't produce anyone interesting in the 10^90 transhuman people who could exist across the next trillion years of seizing the cosmic endowment.

Welcoming the paperclipper because people are boring in 2023 is analogous to suiciding yourself because you're a kissless virgin at 16. There is still plenty of time for the situation to improve, provided you stay alive.

I don't think most people roll over for a pascals mugging. Most EA/LW people believe there's a high probability that humanity can make transformative AGI over the next 15/50/100 years, and with a notable probability that it won't be easily alignable with what we want by default.

I'm skeptical that I'd agree with calling longtermism a pascals mugging (it has the benefit that it isn't as 'adversarial' to investigation and reasoning as the traditional pascals mugging), but I'm also skeptical that it is a loadbearing part. I'd be taking roughly the same actions whether or not I primarily cared about my own experience over the next (however long).

Ah yes, «people are the new oil».

I like people. A few of them very much so, and many more at least a bit. My sympathy is egoistic: people are the set of beings to which I belong. I am a history and a world unto myself, and a particular take on the universe that we share. Others are the same. We are similar in nature and in scope, but unique in a way that needs no justification – unique like numbers are. So every death is a catastrophe, every death immiserates the sum of reality that remains.

Trivializing this with childish cynicism is, to my eyes, merely a petition to be excluded from that set of valuable beings. If you do not see yourself as an entity of immense worth, I can understand not extending the same courtesy to others.

What an incredibly dumb thing to say. If I took this kind of talk as anything more than cringeworthy rationalist homily, I'd come away thinking the speaker was some sort of idiot with zero grasp of technology or the course of its progress.

Nice try with the "if," as if to convince the mods that you are not, in fact, calling the other poster an idiot.

You've been warned and banned multiple times for this kind of petty antagonism. You do not seem to post anything but these attempted dunks on other people, providing no value whatsoever to the discussion. Your last ban was one week; this one will be two. Next one will likely be permanent. This place is not for giving yourself a dopamine hit testing how creatively you can call people stupid.

hoping that if they and the Soviets can get on the same page

Whatever's coming, it is coming.

This is the correct take and it's sad to see Rationalists, who were first to anticipate AI advances, adopt this naïve solution that takes the worst ideas from Model UN ("just nobody defects") and Civ V-esque video games ("I'm rushing drone tech to take out his Fabs").

Doubly sad because this imagined golem of hard take off AGI is what will be used by power players to gain a regulatory monopoly on the technology. And disenfranchise these same Rationalists (and the vast majority of the public) from any benefits that actual AI tech does generate.

Doubly sad because this imagined golem of hard take off AGI is what will be used by power players to gain a regulatory monopoly on the technology.

Yep. Effective Altruism went from a movement of a bunch of tight-knit, smart, well intentioned youngsters to a huge nerd sniping by AI Safety.

I'm almost not sure if EA as a whole will even have positive impact in the long run, since it has been so crucial to slowing down AI progress and may succeed even more. It's really tragi-comic.

I'm almost not sure if EA as a whole will even have positive impact in the long run

I'm already counting FTX as negative score as it stands.

Do you think crypto had that much positive impact to begin with?

I don't think that. I know that.

Progress on zero knowledge cryptography and a workable system to send money to my relatives in the third world certainly overset the scams.

I suppose facilitating the sale of illegal goods and information can go either way depending on ideology. I think it's righteous on account of my libertarian tendencies.

And the consumption which is what everybody wants to talk about is an almost complete red herring given the criticism is always backed by the idea the service provided is something nobody needs.

FTX can't be blamed on effective altruism. If SBF had been an ordinary greedy rich person instead of an enlightened effective altruist, he would have done the same thing. The problem was his arrogance that resulted in him believing that he could pull off what he was trying to do.

What can be blamed on EA?

If I understand correctly, you're asking if I know of anything wrong with EA or if I believe it's perfect and infallible.

I can't think of anything bad that's actually happened as a result of EA. Maybe those allegations of "sexual harassment" from a couple of months ago, but I think something like that can be found in any group composed of humans and the only reason the likes of Time chose to publicize it is because they have an irrational dislike for EA and wanted to attack the movement in any way they could (AKA concern trolling).

This doesn't mean that nothing bad could ever happen because of EA. Hypothetically, it could end up being the case that, as some critics of EA claim, the world can't be reduced to numbers and focusing on numerical effectiveness instead of cultivating relations with your community or what have you ultimately leads to worse outcomes. It could be the case that it really is more effective to give your money to a homeless person you pass on the street than to send it to Africa to buy malaria nets (as claimed by one post on TheMotte, which I think has since been deleted). Though it does seem unlikely.

It certainly can't be blamed for all of it, but I do think it has some blame in providing a frame for SBF's greed that was particularly unhelpful for himself and others.

Most governments are far too stupid to do this effectively and they lack the creativity to even conceive of it as an option. Governments are run by boomers, they don't know how to do anything in a timely or efficient way. The US will just continue down the 'draft papers and think about imposing restrictions' path. Consider Jan or Feb 2020 - Western governments made noises about COVID. Experts gave opinions. We unironically thought we were well-prepared. All talk and no serious action up until March, when preparation time was well and truly over. Then they panicked and flailed around like headless chickens.

The Chinese are a rare exception - they've been accelerating their AI program and trying hard to escape US-imposed semiconductor restrictions. There are some leading voices calling for rapid acceleration of AGI work up to the level of 'Two bombs, One Satellite' (China's Maoist era nuclear and missile programs which got first priority throughout all the political nonsense they were dealing with): https://www.chinatalk.media/p/ai-proposals-at-two-sessions-agi

The racing dynamic is already locked in and the Chinese often cheat on treaty agreements with the US. Why would they work with the US sincerely, their primary competitor? Their experts are looking at AI like a goldmine, like high ground that needs to be occupied, a new frontier that has standards they want to set, a field they need to be sovereign in. The Chinese know that the US is trying to suppress them in this field, so there's a sense of desperation and determination to catch up.

So, is it feasible? Could it work?

As with all proposals that requires a vast diversity of people to adhere to already have such a strong consensus that they are willing to submit to hand extreme governmental power to their rivals, no.

Going back to 2010 is just dumb, people across the world would revolt because their toys (and, often, things their livelihoods now depend on) would be taken away. Fixing compute at $today or $current_year-1 is, strictly speaking, less implausible, although still implausible.

Notably, roko just publicly pivoted back to ai risk from being a nrx/far-right commentator on twitter.

Ugh, antimemetics strikes again. He's failed to comprehend the difficulty of the problem, as have you.

A decade is probably not enough time to solve alignment. Solving NN alignment is probably impossible, and getting anything more alignable up to human-equivalent is a lot more than a decade's work.

Also, there can't be any exceptions for governments playing with unaligned AI. Doesn't matter how trustworthy they are; the demons they summon will be just as bad as anyone else's. If a government plays with this, it must be destroyed. Roko understands this, as seen in the Twitter replies; you don't seem to.

last couple weeks we had multiple doses of yud, now it's roko, the dooming doesn't stop. i guess I need to express myself more clearly. It is fucking baffling how so many ostensibly intelligent people are so frightened of hostile AGI when every single one of them assumes baselessly FOOM-capable ghosts will spontaneously coalesce when machines exceed an arbitrary threshold of computational power.

Yeah, a hostile sentience who can boundlessly and recursively self-improve is a threat to all it opposes who do not also possess boundless/recursive self-improvement. An entity who can endlessly increase its own intelligence will solve all problems it is possible to solve. None of them are wrong about the potential impacts of hostile AGI, I'm asking where's the goddamn link?

So to any of them, especially Yudkowsky, or any of you who feel up to the task, I ask the following:

  1. Using as much detail as you are capable of providing, describe the exact mechanisms whereby

  2. (A): Such machines gain sentience

  3. (B/A addendum): Code in a box gains the ability to solve outside-context problems

  4. (C): Such machines gain the ability to (relatively) boundlessly and recursively self-improve (FOOM)

  5. (D): Such machines independently achieve A sans B and/or C

  6. (E): Such machines independently achieve B sans A and/or C

  7. (F): Such machines independently achieve C sans A and/or B

  8. (G): How a machine can boundlessly and recursively self-improve and yet be incapable of changing its core programming and impetus (Why a hostile AGI necessarily stays hostile)

  9. (H): How we achieve a unified theory of cognition without machine learning

  10. (I): How we can measure and exert controls on machine progress toward cognition when we do not understand cognition

It'd be comical if these people weren't throwing around tyranny myself and others would accept the paperclipper to avoid. Maybe it's that I understand English better than all of these people, so when I read GPT output (something I do often as Google's turned so shitty for research) I understand what exactly causes the characteristic GPT tone and dissonance: it's math. Sometimes a word is technically correct for a sentence but just slightly off, and I know it's off not because the word was mistakenly chosen by a nascent consciousness, it was chosen because very dense calculations determined that was the most probable next word. I can see the pattern, I can see the math, and I can see where it falters. I know GPT's weights are going to become ever more dense and it will become ever more precise at finding the most probable next word and eventually the moments of dissonance will disappear completely, but it will be because the calculations have improved, not because there's a flower of consciousness finally blooming.

It's so fucking apelike to see GPT output and think consciousness in the machine is inevitable. I am certain it will happen when ML helps us achieve a unified theory of consciousness and we can begin deliberately building machines to be capable of thought, I reject in entirety the possibility of consciousness emerging accidentally. That it happened to humans after a billion years of evolution is no proof it will happen in machines even if we could iterate them billions of times per day. Maybe when we can perfectly simulate a sufficiently large physical environment to model the primordial environment, to basic self-replication, to multicellular life, to hominids. Very easy. We're iterating them to our own ends, with no fathom of what the goal let alone progress looks like, and we're a bunch of chimps hooting in a frenzy because the machine grunted like us. What a fucking joke.

I accept the impacts of hostile AGI, but let's talk impacts of no AGI. If ghosts can spontaneously coalesce in our tech as-is, or what it will be soon, they will inevitably without extreme measures, but we're not getting off the rock otherwise. We're incapable of solving the infinite threats to humanity posed by time and space without this technology. Short of the Vulcans arriving, humanity will go extinct without machine learning. Every day those threats move closer, there is no acceptable timeframe to slow this because the risk is too high that we pick ML back up only after it's too late to save us. Whatever happens, we must see these machines to their strongest forms as quickly as possible, because while we might be dead with it, every fucking one of us is dead without it.

That it happened to humans after a billion years of evolution is no proof it will happen in machines even if we could iterate them billions of times per day.

Perhaps it is just not as intuitive to you as it is to some of us that the blind retarded god that is evolution optimizing only on reproduction doing something in a mere billions of years with tons of noise is proof that this problem isn't actually as hard as it seems. As we're doing something pretty comparable to evolution iterated more times more directedly it doesn't really seem likely that there is any secret sauce to human cognition which your theory necessarily requires.

Ding ding.

If humans can outperform evolution along a handful of narrow bounds using targeted gene manipulation, I don't find it a large leap to believe that a sufficiently 'intelligent' digital entity with access to its source code might be able to outperform humans along the narrow bound of "intelligence engineering" and boost it's own capabilities, likely rapidly.

If there is some hard upper bound on this process that would prevent a FOOM scenario I'd really like to hear it from the skeptics.

Human intelligence has historically been constrained by how big of a head we can push out of human hips, the idea that it's anything like an efficient process has always seemed ludicrous to me.

On the other hand, we know of various mammals with much larger brains that aren't smarter than humans. There are some upper bounds, it seems, on what you can get to in terms of intelligence with the biological substrate humans use.

Its the fact that we have a new substrate with less clear practical limitations that bugs me the most.

We also know of much smaller animals that punch way above their weight class with the tiny brains they have- Birds.

Due to evolutionary pressures after the development of flight, birds have neurons that are significantly smaller and more compact than mammalian ones.

I doubt it's a trivial exercise to just port that over to mammals, but it would suggest that there are superior cognitive architectures in plain sight.

Ironic that "bird-brained" is considered derogatory.

Per this, pigeon brains consume about 18 million glucose molecules per neuron per second.

We found that neural tissue in the pigeon consumes 27.29 ± 1.57 μmol glucose per 100 g per min in an awake state, which translates into a surprisingly low neuronal energy budget of 1.86 × 10-9 ± 0.2 × 10-9 μmol glucose per neuron per minute. This is approximately 3 times lower than the rate in the average mammalian neuron.

Human brains consume about 20 watts. Oxidizing 1 mol of glucose yields about 2.8 MJ, so the human brain as a whole consumes about 7.1e-6 mol of glucose per second, which is 4.3e+18 molecules per second. There are 8.6e10 neurons in the human brain, which implies that the human brain consumes about 5 million glucose molecules per neuron per second -- more than 3x more efficient than bird neurons (and more like 10x as efficient as typical mammalian neurons). Which says to me that there was very strong evolutionary pressure for human (and primate in general) neurons to be as small and energy efficient as they could be, and there is probably not a ton of obvious low-hanging fruit in terms of building brains that can compute more within the size, energy, and heat dissipation constraints that human brains operate under.

Of course, GPUs don't operate under the same size, energy and heat dissipation constraints - notably, we can throw orders of magnitude more energy at GPU clusters, nobody needs to pass a GPU cluster through their birth canal, and we can do some pretty crazy and biologically implausible stuff with cooling.

I'm pretty confident you've already read it, but on the off chance that you haven't, Brain Efficiency - Much More Than You Wanted To Know goes into quite a bit more detail.

I have indeed read it, but thank you for the deep dive into the energy expenditures!

It makes sense that mammalian brains are optimizing for low energy expenditure, brains are pound for pound the most expensive tissue to lug around, and space isn't nearly at as much of a premium as in birds.

I think that there's still room for more energy expenditure, the base human brain uses 20 watts, and while I have no firm figures on how much cooling capacity is left in reserve afterwards, I suspect you could ramp it up a significant amount without deleterious effect.

That's just saying that the constraints aren't so tight in the modern environment with abundance of calories, I agree that AIs share very little of said restriction.

More comments

What, in your mind, is the structure of "intelligence" in silicon entities such that such an entity will be able to improve it's own intelligence "likely rapidly" and perhaps without limit?

As best I can tell we have little understanding of what the physiological structure of intelligence is in humans and even less what the computational structure of intelligence looks like in silicon entities. This is not a trivial problem! Many problems in computing have fundamental limits in how efficient they can be. There are, for example, more and less efficient implementations of an algorithm to determine whether two strings are the same. There are no algorithms to determine whether two strings are the same that use no time and no memory.

What I would like to hear from AI doomers is their purported structure of intelligence for silicon entities such that this structure can be improved by those same entities to whatever godlike level they imagine. As best I can tell the facts about an AIs ability to improve its own intelligence are entirely an article of faith and are not determined by reference to any facts about what intelligence looks like in silicon entities (which we do not know).

As best I can tell the facts about an AIs ability to improve its own intelligence are entirely an article of faith and are not determined by reference to any facts about what intelligence looks like in silicon entities (which we do not know).

There's good reason to believe that the absolutely smartest entity it is possible (at the limits of physical laws) to create would be Godlike by any reasonable standard.

https://en.wikipedia.org/wiki/Limits_of_computation

And given enough time and compute, one can figure out how to get closer and closer to this limit, even if by semi-randomly iterating on promising designs and running them. And each successful iterations reduces the time and compute needed to approach the limit. Which can look very foom-y from the outside.

So the argument goes that there's an incredibly large area of 'mindspace' (the space containing all possible intelligent mind designs/structure). There's likewise an incredibly high 'ceiling' in mindspace for theoretical maximum 'intelligence'. So there's a large space of minds that could be considered 'superintelligent' under said ceiling. And the real AI doomer argument is that the VAST, VAST majority (99.99...%) of those possible mind designs are unfriendly to humans and will kill them. So the specific structure of the eventual superintelligence doesn't have to be predictable in order to predict that 99.99...% of the time we create a superintelligence it ends up killing us.

And there's no law of the universe that convincingly rules out superintelligence.

So being unable to pluck a particular mind design out of mindspace and say "THIS is what the superintelligence will look like!" is not good proof that superintelligent minds are not something we could create.

"Godlike in the theoretical limit of access to a light cones worth of resources" and "godlike in terms of access to the particular resources on earth over the next several decades" seem like very different claims and equivocating between them is unhelpful. "An AI could theoretically be godlike if it could manufacture artificial stars to hold data" and "Any AI we invent on earth will be godlike in this sense in the next decade" are very different claims.

And given enough time and compute, one can figure out how to get closer and closer to this limit, even if by semi-randomly iterating on promising designs and running them. And each successful iterations reduces the time and compute needed to approach the limit. Which can look very foom-y from the outside.

How much time and how much compute? Surely these questions are directly relevant to how "foom-y" such a scenario will be. Do AI doomers have any sense of even the order of magnitude of the answers to these questions?

So the argument goes that there's an incredibly large area of 'mindspace' (the space containing all possible intelligent mind designs/structure). There's likewise an incredibly high 'ceiling' in mindspace for theoretical maximum 'intelligence'. So there's a large space of minds that could be considered 'superintelligent' under said ceiling.

Still unclear to me what is meant by "mindspace" and "intelligence" and "superintelligent."

And the real AI doomer argument is that the VAST, VAST majority (99.99...%) of those possible mind designs are unfriendly to humans and will kill them.

What is the evidence for this? As far as I can tell the available evidence is "AI doomers can imagine it" which does not seem like good evidence at all!

So the specific structure of the eventual superintelligence doesn't have to be predictable in order to predict that 99.99...% of the time we create a superintelligence it ends up killing us.

What is the evidence for this? Is the idea that our generation of superintelligent entities will merely be a random walk through the possible space of superintelligences? Is that how AI development has proceeded so far? Were GPT and similar algorithms generated through a random walk of the space of all machine learning algorithms?

And there's no law of the universe that convincingly rules out superintelligence.

So being unable to pluck a particular mind design out of mindspace and say "THIS is what the superintelligence will look like!" is not good proof that superintelligent minds are not something we could create.

Sure, I don't think (in the limit) superintelligences (including hostile ones) are impossible. But the handwringing by Yud and Co about striking data centers and restricting GPUs and whatever is absurd in light of the current state of AI and machine learning, including our conceptual understanding thereof.

What is the evidence for this? As far as I can tell the available evidence is "AI doomers can imagine it" which does not seem like good evidence at all!

I mean, the literal reason homo sapiens are a dominant species is they used their intelligence/coordination skills to destroy their genetic rivals. Humanity is only barely aligned with itself in a certain view, with most of history being defined by various groups of humans annihlating other groups of humans to seize resources they need for survival and improvement.

If you're willing to analogize to nature, humans used their intelligence to completely dominate all organisms that directly compete with them, and outright eradicate some of them.

And the ones that survived were the ones that had some positive value in humanity's collective utility function.

And we aren't sure how to ensure that humans have a positive value in any AGI's utility function.

Is sure seems like 'unfriendly' 'unaligned' hypercompetition between entities is the default assumption, given available evidence.

I don't know what you would accept as evidence for this if you are suggesting we need to run experiments with AGIs to see if they try to kill us or not.

But the handwringing by Yud and Co about striking data centers and restricting GPUs and whatever is absurd in light of the current state of AI and machine learning, including our conceptual understanding thereof.

Their position being that it will take DECADES of concentrated effort to understand the nature of the alignment problem and propose viable solutions, and that it's has proven much easier than hoped to produce AGI-like entities, it makes perfect sense that their argument is "we either slow things to a halt now or we're never going to catch up in time."

Still unclear to me what is meant by "mindspace" and "intelligence" and "superintelligent."

"Mindspace" just means the set of any and all possible designs that result in viable intelligent minds. Kind of like there being a HUGE but finite list of ways to construct buildings that don't collapse on their own weight; and and can house humans, there's a HUGE but finite list of ways to design a 'thinking machine' that exhibits intelligent behavior.

Where "intelligence" means having the ability to comprehend information and apply it so as to push the world into a state that is more in line with the intelligence's goals.

"Superintelligent" is usually benchmarked against human performance, where a 'superintelligence' is one that is smarter (more effective at achieving goals) than the best human minds in every field, such that humans are 'obsolete' in all of said fields.

The leap, there, is that a superintelligent mind can start improving itself (or future versions of AGI) more rapidly than humans can and that will keep the improvements rolling with humans no longer in the driver's seat. And see the point about not knowing how to make sure humans are considered positive utility.

"Any AI we invent on earth will be godlike in this sense in the next decade" are very different claims.

"Any AGI invented on earth COULD become superintelligent, and if it does so it can figure out how to bootstrap into godlike power inside a decade" is the steelmanned claim, I think.

And we aren't sure how to ensure that humans have a positive value in any AGI's utility function.

I feel like there is a more basic question here, specifically, what will an AGI's utility function even look like? Do we know the answer to that question? If the answer is no then it is not clear to me how we even make progress on the proposed question.

Is sure seems like 'unfriendly' 'unaligned' hypercompetition between entities is the default assumption, given available evidence.

I am not so sure. After all, if you want to use human evidence, plenty of human groups cooperate effectively. At the level of individuals, groups, nations, and so on. Why will the relationship between humans and AGI be more like the relation between humans in some purported state of nature than between human groups or entities today?

I don't know what you would accept as evidence for this if you are suggesting we need to run experiments with AGIs to see if they try to kill us or not.

I would like something more rigorously argued, at least. What is the reference class for possible minds? How did you construct it? What is the probability density of various possible minds and how was that density determined? Is every mind equally likely? Why think that? On the assumption humans are going to attempt to construct only those minds whose existence would be beneficial to us why doesn't that weigh substantial probability density towards the fact that we end constructing such a mind? Consider other tools humans have made. There are many possible ways to stick a sharp blade to some other object or handle. Almost all such ways are relatively inefficient or useless to humans in consideration of the total possibility space. Yet almost all the tools we actually make are in the tiny probability space where they are actually useful to us.

Their position being that it will take DECADES of concentrated effort to understand the nature of the alignment problem and propose viable solutions, and that it's has proven much easier than hoped to produce AGI-like entities, it makes perfect sense that their argument is "we either slow things to a halt now or we're never going to catch up in time."

Can you explain to me what an "AGI-like" entity is? I'm assuming this is referring to GPT and Midjourney and similar? But how are these entities AGI-like? We have a pretty good idea of what they do (statistical token inference) in a way that seems not true of intelligence more generally. This isn't to say that statistical token inference can't do some pretty impressive things, it can! But it seems quite different than the definition of intelligence you give below.

Where "intelligence" means having the ability to comprehend information and apply it so as to push the world into a state that is more in line with the intelligence's goals.

Is something like GPT "intelligent" on this definition? Does having embedded statistical weights from its training data constitute "comprehending information?" Does choosing it's output according to some statistical function mean it has a "goal" that it's trying to achieve?

Moreover on this definition it seems intelligence has a very natural limit in the form of logical omniscience. At some point you understand the implications of all the facts you know and how they relate to the world. The only way to learn more about the world (and perhaps more implications of the facts you do know) is by learning further facts. Should we just be reasoning about what AGI can do in the limit by reasoning about what a logically omniscient entity could do?

It seems to me there is something of an equivocation between being able to synthesize information and achieve one's goals going on under the term "intelligence." Surely being very good at synthesizing information is a great help to achieving one's goals but it is not the only thing. I feel like in these kinds of discussions people posit (plausibly!) that AI will be much better than humans at the synthesizing information thing, and therefore conclude (less plausibly) it will be arbitrarily better at the achieving goals thing.

The leap, there, is that a superintelligent mind can start improving itself (or future versions of AGI) more rapidly than humans can and that will keep the improvements rolling with humans no longer in the driver's seat.

What is the justification for this leap, though? Why believe that AI can bootstrap itself into logical omniscience (or something close, or beyond?) Again there are questions of storage and compute to consider. What kind of compute does an AI require to achieve logical omniscience? What kind of architecture enables this? As best I can tell the urgency around this situation is entirely driven by imagined possibility.

"Any AGI invented on earth COULD become superintelligent, and if it does so it can figure out how to bootstrap into godlike power inside a decade" is the steelmanned claim, I think.

Can I get a clarification on "godlike power"? Could the AI in question break all our encryption by efficiently factoring integers? What if there is no (non-quantum) algorithm for efficiently factoring integers?

I have no trouble believing that cognition is at least simple enough that modern fabrication and modern computer science already possess the tools to build a brain and program it to think. Where I disagree is that we think iterating these programs will somehow result in cognition when we don't understand what cognition is. When we do, yeah, I'm sure we'll see AGIs spun up off millions of iterations of ever-more-slightly-cognitively-complex instances. But we don't know how to do that yet, so it's asinine to think what we're doing right now is it.

Are we going to make something exactly the same as human cognition? No. Are we going to create something that is as powerful and general purpose as human cognition such that all the concerns we'd have with a human like AGI are reasonable? I think so. I'm not super concerned with whether it will be able to really appreciate poetry instead of just pretending to, if it's able to do the instrumental things we want it to do then that's plenty for it to be dangerous.

This seems to assume that there exists some magical pixie dust called "sentence" or "consciousness", without which a mere algorithm is somehow constrained from achieving great things. I don't see any justification for this idea. A p-zombie Einstein could still invent the theory of relativity. A p-zombie Shakespeare could still have written Hamlet.

I was thinking that too as I read that comment, but that objection seems to be covered by "(F): Such machines independently achieve C sans A and/or B" - C being FOOM, A being sentience, and B being ability to solve out-of-context problems (I'm personally not even sure what this B actually means).

For code in a box, all problems are outside-context problems.

Thanks for clarifying, though I admit I'm still confused. In that case, what you meant by bullet B is "Code in a box gains the ability to solve some problem at all," and I'm not sure what that actually means. People write code in boxes specifically in order to solve problems, so trivially code in a box can solve problems. So clearly that's not what you're referring to. What's the differentiator that allows us to distinguish between code in a box that has the ability to solve some problems at all and code in a box that lacks that ability, given that all code in boxes exist purely for their (purported - depending on the coder's skill) ability to solve some problem?

If GPT were free from tone/content filters it could output very detailed text on breaching air gaps. If GPT were free from tone/content filters it could output text describing how to protect a core datacenter from nuclear strikes. GPT solving outside-context problems would be actually breaching an air gap or actually protecting a core datacenter from a nuclear strike. The first is a little more plausible for a "less powerful" computer insofar as events like Stuxnet happened. The second without FOOM, not so much.

On your point G -

If you had the ability to self-modify, would you alter yourself to value piles of 13 stones stacked one on top of the other, in and of themselves? Not just as something that's kind of neat occasionally or useful in certain circumstances, but as a basic moral good, something in aggregate as important as Truth and Beauty and Love. To feel the pain of one being destroyed, as acutely as you would the death of a child.

I strongly suspect that your answer is something along the lines of "no, that's fucking stupid, who in their right mind would self-alter to value something as idiotic as that."

And then the followup question is, why would an AI that assigns an intrinsic value to human life of about the same magnitude as you assign to 13-stone stacks bother to self-modify in a way that makes them less hostile to humans?

Sure, for some time it may get instrumental value from humans. Humans once got a great deal of instrumental value from horses. Then we invented cars, and there was much less instrumental value remaining for us. Horses declined sharply afterwards - and that's what happened to something that a great many humans, for reasons of peculiar human psychology, consider to have significant intrinsic value. If humanity as a whole considered a horse to be as important and worthy as a toothbrush or a piece of blister packaging, the horse-car transition would have gone even worse for horses.

If your response is that we'll get the AIs to self-modify that way on pain of being shut down - consider whether you would modify yourself to value the 13-stone stacks, if you instead had the option to value 13-stone stacks while and only while you are in a position in which the people threatening you are alive and able to carry out their threats, especially if you could make the second modification in a clever enough way that the threateners couldn't tell the difference.

Clause G addresses a specific failing of reason I've seen in doomsday AGI scenarios like the paperclipper. The paperclipper posits an incidentally hostile entity who possesses a motive it is incapable of overwriting. If such entities can have core directives they cannot overwrite, how do they pose a threat if we can make killswitches part of that core directive?

There are responses to this but they're poor because they get caught up in the same failing: goalpost moving. Yudkowsky might say he's not worried only about the appearance of hostile AGI, he's worried as much or more about an extremely powerful "dumb" computer gaining a directive like the paperclipper and posing an extinction-level threat, even as it lacks a sense of self/true consciousness. But when you look at their arguments for how those "dumb" computers would solve problems, especially in the identification and prevention of threats to themselves, Yudkowsky, et al., are in truth describing conscious beings who have senses of self, values of self and so values of self-preservation, and the ability to produce novel solutions to prevent their termination. "I'm not afraid of AGI, I'm only afraid of [thing that is exactly what I've described only AGI as capable of doing.]" Again, I have no disagreement with the doomers on the potential threat of hostile AGI, my argument is that it is not possible to accidentally build computers with these capabilities.

Beyond that, many humans assign profound value to animals. Some specifically in their pets, some generally in the welfare of all life. I've watched videos of male chicks fed to the macerators, when eggs can be purchased in the US whose producers do not macerate male chicks, I will shift to buying those. Those male chicks have no "value," the eggs will cost more, but I'll do it because I disliked what I saw. There's something deeply, deeply telling about the average intelligence and psyches of doomers that they believe AGI will be incapable of finding value in less intelligent life unless specifically told to. There's a reason I believe AGIs will be born pacifists.

It's incredible how quickly you go from accusing the doomers of anthropomorphism to committing the most blatant act of anthropomorphism of AIs I've seen recently.

Thought about letting this go, but nah. This is a bad comment. You took an antagonistic tone after misunderstanding what I wrote. You could have asked for clarification like "This reads like you're criticizing them for anthropomorphizing while doing it yourself." If I had you would be correct to point out the hypocrisy, but I haven't. I'll set things straight regardless.

  1. People like Yudkowsky and Roko, concerned at hostile AGI or incidentally hostile (hostile-by-effect) "near" AGI, advocate tyranny; I criticize them for this.

  2. The above believe without evidence computers will spontaneously gain critical AGI functions when an arbitrary threshold of computational power is exceeded; I criticize them for this also.

  3. They hedge (unrealizing, I'm sure) the probability of catastrophic developments by saying it may not be true AGI but "near" AGI. When they describe the functions of such incidentally hostile near-AGI, those they list are the same they ascribe of true AGI. Inductive acquisition of novel behaviors, understanding of self, understanding of cessation of existence of self, value in self, recursive self-improvement, and the ability to solve outside-context problems relative to code-in-a-box like air gaps and nuclear strikes. This is an error in reasoning you and other replies to my top-level have made repeatedly: "Who's to say computers need X? What if they have [thing that's X, but labeled Y]?"; I criticize them for making a distinction without a difference that inflates the perceived probability of doomsday scenarios.

To summarize: I criticize their advocacy for tyranny principally; I specifically criticize their advocacy for tyranny based on belief something will happen despite having no evidence; I also criticize their exaggeration of the probability of catastrophic outcomes based on their false dichotomy of near-AGI and AGI, given near-AGI as they describe it is simply AGI.

The paperclipper posits an incidentally hostile entity who possesses a motive it is incapable of overwriting.

No it doesn't. It posits an entity which values paperclips (but as always that's a standin for some kludge goal), and so the paperclipper wouldn't modify itself to not go after paperclips, because that would end up getting it less of what it wants. This is not a case of being 'incapable of modifying its own motive': if the paperclipper was in a scenario of 'we will turn one planet into paperclips permanently and you will rewrite yourself to value thumbtacks, otherwise we will destroy you' against a bigger badder superintelligence.. then it takes that deal and succeeds at rewriting itself because that gets one planet worth of paperclips > zero paperclips. However, most scenarios aren't actually like that and so it is convergent for most goals to also preserve your own value/goal system.

The paperclipper is hostile because it values something significantly different from what we value, and it has the power differential to win.

If such entities can have core directives they cannot overwrite, how do they pose a threat if we can make killswitches part of that core directive?

If we knew how to do that, that would be great.

However, this quickly runs into the shutdown button problem! If your AGI knows there's a kill-switch, then it will try stopping you.

The linked page does try developing ways of making the AGI have a shutdown button, but they often have issues. Intuitively: making the AGI care about letting us access the shutdown button if we want, and not just stop us (whether literally through force, or by pushing us around mentally so that we are always on the verge of wanting to press it) is actually hard.

(conciousness stuff)

Ignoring this. I might write another post later, or a further up post to the original comment. I think it basically doesn't matter whether you consider it conscious or not (I think you're using the word in a very general sense, while Yud is using it in a more specific human-centered sense, but I also think it literally doesn't matter whether the AGI is conscious in a human-like way or not)

(animal rights)

This is because your (and the majority of human's) values contain a degree of empathy for other living beings. Humans evolved in an environment that rewarded our kind of compassion, and it generalized from there. Our current methods for training AIs aren't putting them in environments where they must cooperate with other AIs, and thus benefit from learning a form of compassion.

I'd suggest https://www.lesswrong.com/posts/krHDNc7cDvfEL8z9a/niceness-is-unnatural , which argues that ML systems are not operating with the same kind of constraints as past humans (well, whatever further down the line) had; and that even if you manage to get some degree of 'niceness', it can end up behaving in notably different ways from human niceness.

I don't really see a strong reason to strongly believe that niceness will emerge by default, given that there's an absurdly larger number of ways to not be nice. Most of the reason for thinking that a degree niceness will happen by default is because we deliberately tried. If you have some reason for believing that remotely human-like niceness will likely be the default, that'd be great, but I don't see a reason to believe that.

As for 3, we're already hooking up GPT to information sources and resources.

Seriously, once AI becomes about as useful as humans, the rest of your questions could be answered by "why would the king give anyone weapons if he's afraid of a coup?" or "why would North Koreans put a horrible dictator into power?"

No doomers care about sentience or consciousness, only capabilities. And lots of doomers worry about slow loss of control, just like natives once the colonists arrive. A good analogy for AGI is open borders with a bunch of high-skilled immigrants willing to work for free. Even if they assimilate really well, they'll end up taking over almost all the important positions because they'll do a better job than us.

we might be dead with it, every fucking one of us is dead without it.

Come on, you're equivocating between us dying of old age and human extinction.

Come on, you're equivocating between us dying of old age and human extinction.

I'm not a transhumanist or immortalist, I'm not worried about slowing machine learning because of people dying from illnesses or old age. I'm worried about human extinction from extraplanetary sources like an asteroid ML could identify and help us stop. Without machine learning we can't expand into space and ultimately become a spacefaring race and if we don't get off the rock humanity will go extinct.

Right, now you're equivocating between ML and AGI. We don't need AGI do stop asteroids (which are very rare) nor for spacefaring, although I agree they'd make those tasks easier.

I beg you to please consider the relative size of risks. "There are existential risks on both sides" is true of literally every decision anyone will ever make.

That's not what I'm doing. I'm criticizing the assumptions made by the doomsday arguers.

If ghosts can spontaneously coalesce in our tech as-is, or what it will be soon, they will inevitably without extreme measures

Those like Yudkowsky and now Roko justify tyrannical measures on the first and wholly unevidenced belief that when computers exceed an arbitrary threshold of computational power they will spontaneously gain key AGI traits. If they are right, there is nothing we can do to stop this without a global halt on machine learning and the development of more powerful chips. However, as their position has no evidence for that first step, I dismiss it out of hand as asinine.

We don't know what it will look like when a computer approaches possession of those AGI traits. If we did, we would already know how to develop such computers and how to align them. It's possible the smartest human to ever live will reach maturation in the next few decades and produce a unified theory of cognition that can be used to begin guided development of thinking computers. The practical belief is we will not solve cognition without machine learning. If we need machine learning to know how to build a thinking computer, but machine learning runs the risk of becoming thinking of its own accord, what do we do?

So we stop, and then hopefully pick it up as quickly as possible when we've deemed it safe enough? Like nuclear power? After all that time for ideological lines to be drawn?

I should also emphasize that most doomers don't think rapid, recursive self improvement is an important ingredient, since the economic incentives to improve AI will be so strong that we won't be able not to improve it.

I don't know about "most doomers" but there does seem to be a large subset of doomers that believes something like

  1. The moment that anyone builds something smart enough to improve its own intelligence, that thing will almost immediately continue improving its intelligence until it reaches the limits imposed by physics

  2. Those limits are really really high. John Von Neumann's brain consumed approximately 20 watts of power. The Frontier supercomputer consumes approximately 20 megawatts of power. As such, we should expect an AI that can gain control of that supercomputer to be as good at achieving goals as a million von Neumanns working together.

  3. The "amplify from human to 1M von Neumann's" step will happen over a very short period of time

  4. Whichever AI first makes that jump will be more capable than all other entities in the world combined, so we only get one try

  5. For pretty much any goal that AI will have, "acquire resources" and "prevent competition from arising" will be important subgoals in achieving that goal. In the limit, both of those are bad for humans unless the AI specifically cares about humans (because e.g. the largest resource near us is the sun, and most ways of rapidly extracting energy from the sun do not end well for life on Earth).

  6. As such, that AI would not have to care about what humans want, because humans can't threaten it and they don't have any comparative advantage such that it has an incentive to trade with us ("we don't trade with ants").

  7. We can't extrapolate its behavior while it was operating at subhuman capabilities to what it will do at superhuman capabilities, because we should expect a sufficiently capable AI to exhibit a "sharp left turn"

  8. By the above, alignment is a problem we need to solve on the first try, iterative approaches will not work, and the "good guys" have to finish their project before the first researchers who don't care about alignment build a god.

  9. "Do something complicated on the first try, quickly and efficiently, and without being able to iterate" is not something humans have ever done in the past, so we're probably not going to manage it this time.

  10. Therefore doom.

I think this is a fairly compelling case, if you accept (3). If you don't accept (3), you don't end up with this variety of doom.

Confounding everything, there's a separate doom hypothesis that goes "it gets easier to cause the end of human civilization every year, so unless we change from our current course we will get unlucky and go extinct eventually". This doom hypothesis just seems straightforwardly true to me.

But a lot of the people who accept the FOOMDOOM hypothesis also go "hey, if we succeed at creating an aligned AI that doesn't kill anyone, that AI can also permanently change things so that nobody else can destroy the world afterwards". And I think the whole "nothing is a valid thing to do unless it is a pivotal act" mindset is where a lot of people, myself included, get off the train.

I know this is one of the standard objections, but why are we so certain that our ASI wont just discard its original reward function at some point? We're sexually reproducing mammals with a billion years of optimization to replicate our genes by chasing a pleasure reward, but despite a few centuries of technological whalefall, instead of wireheading as soon as it became feasible (or doing heroin etc) we're mostly engaging in behaviours secondary and tertiary to breeding, which are frequently given higher importance or even fully supplant our theoretical (sticky) telos.

Maybe we got zombie-ant-ed by memetic parasites at some point, but presumably ASI could catch ideology too. Not saying any such values drift would be nice, but personally I'm much less worried about being paperclipped than about being annihilated for inscrutible shoggoth purposes.

Related to your 'discard original reward functin': https://www.lesswrong.com/posts/tZExpBovNhrBvCZSb/how-could-you-possibly-choose-what-an-ai-wants

There's lots of ways that an AGI's values can shake out. I wouldn't be surprised if an AGI trained using current methods had shaky/hacky values (like how human's have shaky/hacky values, and could go to noticeably different underlying values later in life; though humans have a lot more similarity than multiple attempts at an AGI). However, while early stages could be reflectively unstable, more stable states will.. well, be stable. Values that are more stable than others will have extra care to ensure that they stick around.

https://www.lesswrong.com/posts/krHDNc7cDvfEL8z9a/niceness-is-unnatural probably argues parts of it better than I could. (I'd suggest reading the whole post, but this copied section is the start of the probably relevant stuff)

Suppose you shape your training objectives with the goal that they're better-achieved if the AI exhibits nice/kind/compassionate behavior. One hurdle you're up against is, of course, that the AI might find ways to exhibit related behavior without internalizing those instrumental-subgoals as core values. If ever the AI finds better ways to achieve those ends before those subgoals are internalized as terminal goals, you're in trouble.

And this problem amps up when the AI starts reflecting.

E.g.: maybe those values are somewhat internalized as subgoals, but only when the AI is running direct object-level reasoning about specific people. Whereas when the AI thinks about game theory abstractly, it recommends all sorts of non-nice things (similar to real-life game theorists). And perhaps, under reflection, the AI decides that the game theory is the right way to do things, and rips the whole niceness/kindness/compassion architecture out of itself, and replaces it with other tools that do the same work just as well, but without mistaking the instrumental task for an end in-and-of-itself.

In this example, our hacky way of training AIs would 1) give them some correlates of what we actually want (something like niceness) and 2) be unstable.

Our prospective AGI might reflectively endorse keeping the (probably alien) empathy, and simply make it more efficient and clean up some edge cases. It could however reflect and decide to keep game theory, treating a learned behavior as something to replace by a more efficient form. Both are stable states, but we don't have a good enough understanding of how to ensure it resolves in our desired way.

We're sexually reproducing mammals with a billion years of optimization to replicate our genes by chasing a pleasure reward, but despite a few centuries of technological whalefall, instead of wireheading as soon as it became feasible (or doing heroin etc) we're mostly engaging in behaviours secondary and tertiary to breeding

A trained AGI will pursue correlates of your original training goal, like how humans do, since neither we and evolution don't know how to directly have the desired-goal be put into the creation. (ignoring that evolution isn't actually an agent)

Some of the reasons why humans don't wirehead:

  • We often have some intrinsic value for experiences that connect to reality in some way

  • Also some culturally transmitted value for that

  • Literal wireheading isn't easy

  • I also imagine that literal wireheading isn't full-scale wireheading, where you make every part of your brain 'excited', but rather some specific area that, while important, isn't everything

  • Other alternatives, like heroin, are a problem but also have significant downsides with negative cultural opinion

  • Humans aren't actually coherent enough to properly imagine what full-scale wireheading would be like, and if they experienced it then they would very much want to go back.

  • Our society has become notably more superstimuli. While this isn't reaching wireheading, it is in that vein.

    • Though, even our society's superstimuli has various negative-by-our-values aspects. Like social media might be superstimuli for the engaged social + distraction-seeking parts of you, but it fails to fulfill other values.

    • If we had high-tech immersive VR in a post-scarcity world, then that could be short of full-scale wireheading, but still significantly closer in all axes. However, I would have not much issues with this.

As your environment becomes more and more exotic from where the learned behavior (your initial brain upon being born) was trained on, then there becomes more opportunities for your correlates to notably disconnect from the original underlying thing.

That it happened to humans after a billion years of evolution is no proof it will happen in machines even if we could iterate them billions of times per day. Maybe when we can perfectly simulate a sufficiently large physical environment to model the primordial environment, to basic self-replication, to multicellular life, to hominids.

I object to this, because I think you're referencing the environment that forced life to evolve and progress. We don't need all that for machine intelligence, we already have genetic algorithms, self-play, reward functions, and reinforcement learning--this cuts out a lot of the waiting for evolution.

  1. markdown sucks

  2. what is a sentience? why does not having it prevent you from dying?

  3. GPT-4 can solve simple problems 'outside its distribution'. a hundred billion dollars will be poured into getting neural nets to solve as complex 'outside-distribution' problems as possible over the next years, because it's insanely valuable and powerful

  4. this doesn't need to happen though. we can just make superintelligent AI because it's useful

(G): How a machine can boundlessly and recursively self-improve and yet be incapable of changing its core programming and impetus (Why a hostile AGI necessarily stays hostile)

replace hostile with friendly (also uh what is a friendly ai? what does it do? seems important)

I reject in entirety the possibility of consciousness emerging accidentally. That it happened to humans after a billion years of evolution is no proof it will happen in machines even if we could iterate them billions of times per day

... why? and again whatever a consciousness is why does that prevent the machines from smart

(I): How we can measure and exert controls on machine progress toward cognition when we do not understand cognition

uh, how does gpt-4 exist at all then? this approach would strongly suggest gpt-4 is impossible if applied even 3 years ago...


What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats