Shirayuki2
new account of Shirayuki, lost old password
User ID: 4180
Like if the "enemy" was "war" then one might naively believe that creating an army is joining the "enemy".
The difference is that "war" is a concept and "AGI" is, at least theoretically, being physically manifested. If the enemy is "the army about to invade" then you probably don't want to join the enemy army to convince them to turn around.
Given the infinity of total annihilation any qaly calculation is swamped by even a tiny amount of leverage on probability of averting disaster
This is the exact logic that I am saying leads to very bad outcomes. If you believe that the stakes are the infinity of total annihilation, that total annihilation is imminent, and that any leverage thus has infinite value, then you can justify doing anything to gain even miniscule leverage, on nothing but the strength of your own belief. The anime character draining the lives of everyone across the world because they believe that only they can bring the world to salvation is not the protagonist of the story.
This is of course vague because you didn't yourself outline what these structural elements are. Seems like some vague notion of the necessity of action to avoid catastrophe.
I think I have outlined my case well enough.
Adherents of both ideologies believe in an imminent Event, with infinitely high stakes attached to the Event, that they are the only ones capable of making sure the Event leads to utopia, and that at least in the case of Marxism, this set of beliefs lead to the justification of many horrible acts that did not, in fact, meaningfully facilitate a better outcome of the Event. As far as I know, this description does not apply to liberal or capitalistic thought.
Shirayuki2
3mo ago
It depends what you count as first principles and Marxism; by the nature of such a project, it's debatable that whatever comes out the other end can really be called "Marxism".
The two closest, depending on how you view it, would probably be one of
Analytical Marxism, which attempted to drop the LTV and the Hegelian dialectic influences, among other things, in order to explain exploitation and class relations via analytic philosophy and the social sciences. Arguably some of Analytic Marxism made it into modern social democratic thought, although realistically it wasn't particuarly influential in and of itself.
Socialism with chinese characteristics, which took the opposite approach of nominally retaining the LTV, but in practice adding exceptions wide enough to drive a bus through, and considering the remaining contradictions as just another part of the dialectical process, dropping LTV for all practical purposes.
For example, SCC uses the idea that as value and material wealth are distinct concepts in Marxist thought, acknowledgement that capital and market signals are necessary for the creation of material wealth is not a strict violation of the LTV, while Three Represents attempted to broadly redefine "labor" as including "everything that capitalists do" in order to allow capitalists into the CCP.
who is exactly is this criticism levied against
The two most prominent representatives of AI Safety are
-
Eliezer Yudkowsky, who as below commentators have noted, has advocated in Time for a world government to seize power and prevent the development of AI via violence, even to the point of bombing data centres and triggering nuclear war. He was one of the primary inspirations for Altman and Musk to found OpenAI, and indirectly Amodei et al to found Anthropic; organizations dedicated to pushing forth AI capabilities, directly against AI safety's original goal.
-
Scott Alexander, who has recently put his support towards AI 2040, a policy proposal that advocates for the creation of an (effective) global world government to nationalize the means of compute throughout the world, in order to prevent un-authorized AI development.
Aren't the guys bombing data centers aligned with your beliefs?
I would say it is a matter of epistemological confidence; I agree, in the abstract, that the hypothetical Event where AGI is created is likely to have poor outcomes, and that if possible, averting the Event would be a good idea.
The problem is with people who believe that AGI is likely to be created imminently soon and in the near-certainty of existentially dangerous outcomes; this line of thinking means you can justify advocating for low-probability policies such as seizing world compute and triggering global war, with a certainty of having dangerous outcomes, because the stakes otherwise are so imminent and so immense.
Even though I directionally align with the belief that AGI is Bad, I think it is far more likely that such proposals are much more likely to cause "mundane" harms like increased governmental repression and/or reduced technological innovation, without having a particularly high likelihood of averting the Event, and quite possibly making it more imminent and/or much worse.
Shirayuki2
3mo ago
I would say GOFAI was a foundational premise of the initial LW view of AI safety; there used to be a lot of concern about such things as basement AGI and FOOM that now seem a bit silly if compute scaling is what leads to AGI, in the same way that modern Marxists need to contort themselves around LTV, or drop it altogether and re-invent Marxism from first principles.
I agree with ThenElection that under the LLM/scaling paradigm, the orthogonality thesis is probably the better structural parallel to LTV as it was understood in the 19th century, in terms of a open conjecture that leads to inevitable collapse if it is true. Perhaps LLM's will disprove the orthogonality thesis in the same way that marginalism disproved LTV, and in a hundred years we'll be viewing orthogonality the same way we now do LTV.
Shirayuki2
3mo ago
at least the labs are full of safety people who might be able to prioritize alignment work
"Might" is a weak word. Historically, "join your enemies to reform them from within" has pretty much never achieved the desired aims; it seems straightforwardly true that Moloch has won in terms of incentives to push capabilities and accelerate.
the alternative was optimistically we get a year or so more time but no influence
I think if you accept the premise that the world is truly doomed on such a short timescale, then having a few more years is better, actually, than trading billions of QALY's for some nebulous idea of influence that has had no empirical benefit.
early liberal, proto-capitalist, thought also shared these structural elements.
Perhaps. It would be easier to engage with this if you elaborated on why you think this is the case.
Unless AI takeoff has already occurred, I'm not sure why you're declaring them wrong about this.
I suppose you're correct, although it is my impression that ~everyone who believes in short AI timelines / AI safety regulation now believes in scaling compute being the way to AGI. Even non-LLM believers are primarily looking for new ways to apply the bitter lesson, and the focus of AI safety regulation is on trying to control world compute, which would be completely pointless if Gary Marcus got the final laugh vis a vis symbolic AI.
We appear to be in the foothills of AI safety deployment.
Again, the event which hasn't happened yet
Yeah, this is my point. Empirical evidence is that the sum total of AI safety's achievements is, against their own goals, to accelerate the founding of OpenAI/Anthropic and the AI race dynamic, while having made no progress on their actual goals (not even the authors expect Plan A to realistically happen).
Of course, it's possible they could turn it around and heroically pull humanity into post-scarcity abundance, the same way that Marxism still has time to eventuate the communist revolution (arguably it's kind of sort of happening with the DSA); but if the stakes are infinite, then there's nothing to do but keep going, even if you're going in the wrong direction, which is why getting Pascal's Mugged is so dangerous.
It's a fundamentally different position from Marxism, where we have a whole host of negative examples of communism in action.
This is the point I'm trying to make though; perhaps it was poorly communicated. At the turn of the 20th century, Marxism was in vogue amongst many Western intellectuals, because we had not yet seen what would happen with communism in action.
Right now, we are at the equivalent stage of AI safety, where it is in vogue in certain intellectual circles; we have not yet seen what will happen with AI safety, but it shares many structural similarities to the Marxist movements a hundred years ago.
What specific statements do you disagree with?
Of course I can't prove Russell's teapot non-existent, that some genius MIRI researcher won't end up discovering the ten lines of Python needed to achieve alignment, or that some firebrand communist leader won't be the one that finally unites all the workers of the world. That's what makes these ideologies so seductive; the idea that the potential payoffs to the Event are so great, anything is permitted for a chance to eventuate the good outcomes.
Empirically the two camps of "AI safety" most likely to achieve anything in the foreseeable future are the frontier-lab aligned teams that want regulatory capture on models in order to secure their own positions, and the NIMBY anti data-centre anti-AI never build anything populists; neither of these camps are particularly aligned with the original mission of AI safety, and either of these outcomes succeeding would be very poor outcomes in my view.
Shirayuki2
3mo ago
Neither Dua Lipa not any of the prestigious guests are not going to read these books (and neither any other), not that there is something wrong with it.
While I agree that the library is largely performative, Dua Lipa is one of the few celebrities, perhaps the only celebrity, that actually is a serious reader and literary critic.
While the overlap between her literary interests and mine is fairly small, by all accounts she really is fairly well read and even hosts a legitimately quite intellectual podcast where she interviews various prominent authors.
Shirayuki2
3mo ago
With the recent discussion on AI 2040, I have been thinking about how the modern AI safety movement of the early 21st century almost entirely pattern matches against the Marxist movements of the early 20th century.
They were both largely wrong about what begat capitalism / AI; Marxism was very insistent on labor value theory, and LW used to be very focused on symbolic AI/GOFAI/FOOM. Despite this, both were extremely prescient about the eventual end states of capitalism and AI well before the general public were aware of such concepts; capitalism leading to involution, monopolism, inequality, alienation, overproduction vs AI leading to AGI/ASI/singularity/the end of humanity.
Both movements have a history of having rock-solid beliefs, perhaps somewhat overconfidently, that current developments will lead to an imminent, eschatological Event; the 1848 revolutions, WW1 revolutions, and the Great Depression leading to the imminent collapse of capitalism, vs Deep Blue, AlphaGo and LLM's leading to the imminent creation of AGI.
Many genuinely altruistic, intelligent people find themselves drawn to both movements as they become convinced by the prescience of the movement's thinkers and start believing in the imminence of the Event - Scott himself has talked about such sympathies before.
Such people (the vanguard party / AI safetyists) are convinced that they are the ones that know best, that they are the only people who are clearly forecasting the imminent Event, and that it's their role to save the lumpenproles / permanent underclass who don't understand class consciousness / the exponential from the Event.
They both believe if the Event goes poorly it will end the world, and if it goes well it will lead to utopic abundance; hence assigning infinite stakes to the actions of the vanguard party / AI safety movement.
They both (correctly) identify that Molochian co-ordination problems make regular action incapable of affecting enough change to make the Event go well; hence the only way out is absolute control over the world by those ideologically and culturally aligned, in order to bring about utopia (communist revolution to seize the means of production vs bombing the data centres and seizing the means of compute).
Inevitably, all such movements end up being co-opted by those very same Molochian dynamics, while never having actually done anything to prevent the Event or improve the chances of the Event leading to abundance (Mao, Stalin vs frontier lab regulatory capture and degrowth ideological capture).
Frankly I also hold similar sympathies myself, with regards to both ideologies. It really is true that capitalism, despite being the best known economic system out of a lot of worse economic systems, has been the genesis of enormous amounts of externalities and suffering, and it really is true that if/when AGI is ever eventuated, at very best it is still going to be the end of the world as we know it, and at worst if anyone builds it everyone dies.
And yet neither Marxism nor AI safety has ever achieved anything that might facilitate a more favorable outcome for the Event, whether it truly is imminent or not, and mostly have just made everything worse. One can only hope that AI safety doesn't rack up Marxism's mountain of bodies along the way.
It's not really an IQ thing. There was a lot of research and investment that went into AI from a lot of very smart people after AlphaGo, but essentially all of it didn't contribute to the current progress in LLM's at all.
Even if you did make money from e.g Nvidia or Google it would have been for the completely wrong reasons; I don't think that could really be called a successful prediction.
As far as I know not a single person at the time of AlphaGo was calling that sufficient data + compute would lead to the level of generalization we see in LLM's, compared to the narrow single-game approaches of AlphaGo and AlphaZero that were in vogue at the time.
I agree. I think most people who believe in AI safety are true altruists who believe that as well.
And yet the greatest accomplishment of the AI safety movement is Yudkowsky influencing Altman and Musk to found OpenAI, and indirectly Amodei et al to found Anthropic. AI safety as it exists now is pretty much either working directly at frontier labs to push capabilities or working at external think tanks that are treated as propaganda arms of the frontier labs (e.g METR eval chart and Epoch AI eval charts played a non-negligible part in keeping AI investment going).
The goals of the AI safety movement would frankly have been better achieved if it had never existed in the first place. That's why I think it's a self-defeating movement; if relief comes it will be from the laws of physics or from Moloch acting via capital markets, not any achievement of AI safety.
Shirayuki2
3mo ago
Setting aside the plausibility and timelines of this scenario, I've become more and more convinced that the entire concept of AI safety is just incoherent; it's simply impossible that there can ever be safety if we live in a world where AGI can be created via human hands.
Fundamentally, human power structures that broadly benefit everyone exist because one human can only do so much. No government can kill all of their own citizens, because if they kill too many people they will lose their monopoly on violence, and no government can seize all of the wealth of their own citizens, because in the world as it is right now, you need human labor for wealth to have any meaning.
Of course, once transformative AI exists, neither of these conditions remain true; realpolitik demands that humans are made irrelevant once they are rendered uncompetitive, whether wealthy or not, whether powerful or not. I thought this was a good post - No-one escapes the permanent underclass.
The alternative proposals where Yudkowsky suggests to bomb the data centres, and even this maximally optimistic proposal where everyone gets in a circle and sings kumbaya, still both effectively boil down to either "the global totalitarian world government will have absolute power and promise an utopia once you are useless" or "the superintelligence will have absolute power and promise an utopia once you are useless".
Even if alignment is absolutely perfect, the very best outcome possible is having the values of a select few satisfied by an inscrutable god through friendship and ponies or via the Matrix.
Maybe some people think this is the good ending for humanity. I find myself not so sure.
The bull case is that AI research and hardware research is enough of an RLVR amenable problem that drawing lines on graphs will improve capabilities enough so that the AI will be drawing the rest of the general and super intelligent owls.
It's not implausible, but by the very nature of the argument it's not falsifiable or reliably predictable at all.
I agree that the current paradigm seems unlikely to lead to the AGI/ASI these pieces treat as imminent without significant paradigm improvements, which could be 1, 10 or 100 years away for all we know.
I think you really underestimate what goes into running a supermarket with a membership account. There's a reason that Yeltsin's visit to a run of the mill grocery store practically heralded the end of Bolshevism and the USSR.
The sheer scale of the global supply chains from production to transportation to last-mile distribution that go into ensuring hundreds of millions of people can eat practically any food they like from all over the world at only 3% margins are easily one of the greatest achievements of humanity as a whole.
Shirayuki2
3mo ago
What happened to learning multiple languages being infeasible?
Shirayuki2
3mo ago
The bull case for data centers in space is that the bottlenecks for getting data centers constructed are going to be
a) permitting / regulation
b) power generation / storage
Data centers in space have the potential to solve both of these bottlenecks, in exchange for introducing the manyfold engineering challenges of getting them into space (which other posts have already described in great detail).
I'm not entirely sure I believe this thesis, but it's at least fairly plausible to me that the engineering challenges can be overcome while NIMBYism gets ever more dysfunctional and terrestial power generation remains a bottleneck.
It has nothing to do with airstrikes though; any serious nation already can or soon will be able to strike assets in LEO all the same, not to mention that if push comes to shove the owners and operators of data centers in space are very much going to be within the reach of nation states.
https://www.hyperstitionai.com/unslop-results
I came across the results of this contest to generate LLM-written stories, with one of our participants being our very own @self_made_human. I'd be interested in any thoughts you had on participating in the contest as well, if you had any.
My main takeaway after skimming the finalists was that while some of the concepts were interesting, even with a lot of prompting and harness effort, there's just not much difference from the sloppy prose and incoherent writing you get when one-shotting text out of an LLM.
It's hard to tell how much of this is just because writing's not a focus at the LLM labs right now, but the fact that every LLM converges onto similar writing attractor states definitely makes me bearish on LLM's ability to continue to generalize.
Shirayuki2
4mo ago
Amusing to me that magicalkittycat got mass downvoted without substantial rebuttal to his post for expressing what I thought were very reasonable points, but also got an AAQC on the very same post. Not sure what so many people found so objectionable with the original post.
Then why is it that so many students who spend hundreds of hours studying a foreign language in school come out the other end not knowing how to speak them?
Because in monolingual countries with a large population, neither the student not the teacher cares enough to have the student come out with a strong grasp of the foreign language on the other end. In countries where getting good at English or another lingua franca is a practical necessity, you better believe a lot of students are coming out with functional language skills despite many of these students having a fraction of the investment rich Anglo countries do.
Most kids have the capacity to learn a language really easily, because humans are designed to learn to talk. They don't have the capacity to learn two
From "Language is Culture" by Spandrell:
This doesn't make any sense. While obviously there are physical limits in the extreme, there is no evidence that language acquisition is bottlenecked by storage space of all things - if you have a GeForce 256 and a 100TB SSD obviously storage is not going to be the bottleneck in your system.
As Pigeon says bilingualism is very common globally - the fact that many children don't learn multiple languages comes from a lack of incentives rather than a lack of ability. Many people across South America, continental Europe, Africa, and ex-CJK Asia will at least be conversant in a global lingua franca and a native language out of necessity: e.g the Nordic countries and the Dutch are almost all perfectly bilingual English speakers.
From "The Myth of 'They Weren't Ever Taught…'" by Education Realist:
And from "The Numbers Speak: Foreign Language Requirements Are a Waste of Time and Money" by Bryan Caplan
These are indictments against the requirement to learn languages in school, which I agree is useless for unmotivated students, but really this is a general argument against teaching anything in school above basic numeracy and literacy. I broadly agree with Caplan's Case Against Education thesis, but these posts do not support your bailey that secondary language acquisition is not possible or detrimental.
The motte here is that the opportunity costs of language learning are high, which I think is probably correct for English speakers even as a hobbyist polyglot myself, but in the long run everyone is dead anyways. There are much worse and unproductive things that most people do with their lives.
Shirayuki2
4mo ago
Houellebecq was writing about this thirty years ago. The sexual revolution has indeed always drawn sexless men into the extension of the domain of struggle.
So I'm left wondering what sort of wealth tax could be implemented, in a way that doesn't destroy too much value while still being meaningful enough to appease the envious
It's an interesting problem, and I'm convinced this is going to be one of the largest issues facing society in the years to come, but at least I cannot think of any easy solution.
On the first hand, I broadly believe that wealth inequality is not a problem in and of itself - it makes no particular difference that someone else has much more than I do, as long as I have what I want and need myself. To an extent where it is a problem, in that wealth allows for the concentration of power, I've always thought that it was merely a symptom of the immutable fact that increasing technological progress allows for the increased centralization of power. If there was someone capable of expropriating the wealth of the billionaires, it merely means that that someone else has expropriated their power - the government can exert much more control than any private capitalist ever could.
On the second hand, empirically the median person really does not like experiencing wealth inequality, especially in the modern social-media driven age of envy. It's easy enough to say that if everyone was an enlightened buddha, then wealth inequality would not be a problem, but it's much like saying that resource allocation would be easy if everyone simply got along - simply impossible in practice. If it were possible to burn value in order to satisfy people's lizard hindbrains and improve social cohesion, I'm becoming more convinced that this would be a good thing, even if it were not optimal in terms of resource allocation.
On the third hand, it really does not seem like there's any level of reduced inequality that would actually satisfy the median person without ruinous consequences. Complaining about billionaires and trillionaires is a useful Schelling point, but in practice I do not see that "mere" centimillionaires worth 999 million are any less targets of jealousy - in practice most people are just envious of anyone with more than them, while believing that they're all fully deserving of what they themselves have. As others write below, there are many countries which already implement wealth taxes and/or have lower Gini indices than the US, but it doesn't seem to have meaningfully reduced the inherent envy that people feel towards those with more than them at all.
As a response to @magicalkittycat below as well:
An administration official told Axios they do not view other models as national security threats because they do not surpass the bar that Mythos set.
Anything at Mythos level or above would need to go through the administration to ensure the government's national security apparatus is hardened enough, the official added.
It seems that the USG is at least nominally interested in regulating models from the other H-bomb teams as well, and one of the claimed reasons that Anthropic is being targeted is because they're considered to be the ones with the strongest model worthy of regulation. Source.
Perhaps you don't trust the administration when they say this - it's hard to know what the second-hand effects of these expert controls are going to be, and this could certainly have been handled better by both parties. Still, if you're concerned about doom and would prefer increased regulation to try and avert doom, this seems to be at least a potential step towards something better.
In an ideal world we would have a council of LKY clones making maximally technocratic decisions, but we live in the world that we live in and nobody else is going to be stepping in.
Statement on the US government directive to suspend access to Fable 5 and Mythos 5.
A few immediate thoughts:
For one strain of doomer, this is a good thing: the USG showing willingness to directly intervene and cut down the addressable market for capability-pushing models reduces economic incentives to push the frontier at all costs and slows down the race dynamic. Correspondingly, it's going to be a nightmare for the markets: the AI boom is driven by the idea that frontier models are going to be replacing a significant chunk of global labor, and obviously now this is significantly less likely now that frontier models are going to be stuck in Uncle Sam's basement.
For another strain of doomer, this is a horrible thing: really this was fairly clear even with the DOD conflict, but now control of AI is now very firmly a direct White House concern - the fate of the planet and/or the universe is in the end going to be decided by DC bureaucrats and not SF tech nerds.
Companies, especially non-US companies, are going to be rushing to the doors to move their AI workflows off closed-source and onto self-hosted models, another blow for the AI market thesis - the business continuity risks of Anthropic's demonstrated willingness to silently cripple models and USG's demonstrated willingness to now arbitrarily cut off access are simply going to be too much.
Shirayuki2
4mo ago
Money is an abstraction over resource allocation in this context - I agree that production is not zero-sum, but many of the inputs to production are obviously rivalrous; land, power, raw resources, construction workers and chip fab allocation are all finite and rivalrous to name a few.
If you're just trying to be edgy about lack of state capacity being why shit doesn't get built then I agree, in many cases resource allocation is not the limiting factor as to why no progress is made, but it's farcial to claim that resource allocation isn't the limiting factor for anything that might directly improve people's lives.
Implicit in such claims is that "we should use the capabilities army to stop anyone else from mustering an army", hence turning into the bandits that you were so afraid of in the first place.
Much of the global south would disagree with that assessment. Still, the obvious differences between HEU and compute is that there is no justifiable civilian use for HEU, and the existential risk of HEU is plainly obvious, whereas compute is dual-use with vast civilian applications, while the existential risk is speculative at best. Any serious attempt to control compute will be useless at best, and lead to eternal tyranny / WW3 at worst.
This is why I specified the early 20th century; there was a legitimate widespread belief amongst many 20th century intelligentsia that the fall of capitalism was imminent around the WW1 revolutions and the Great Depression, when it really did look like the contradictions of capitalism were coming to roost.
No, it is indeed broadly true of 20th century Marxism. Traditional Marxism considers the communist revolution to be a final, eschatological event capable of ending the class struggle, and with the end of class struggle an end to the suffering, exploitation and conflict plaguing humanity for the rest of time. Lenin and Trotsky were both adherents to this school of thought, where they believed that they were the ones at the vanguard to the most important moment in all of history, precisely because of the immensity of the stakes.
Sure, I will concede that the Yud school of thought is less about creating utopia and more about avoiding infinite dystopia, while the Scott school is talking about ushering in utopic superintelligence by 2040. The point is that AI safetyists act like Cassandra in having very strong beliefs where they are the only ones that are clearly forecasting the imminent disaster and that the rest of society is badly mistaken; being over-confident about like, who wins the World Cup has few consequences unless you gamble too much on Kalshi, but it is very dangerous when you also believe that the stakes of your inside view are functionally infinite.
Well, 20th century Marxism directly lead to the USSR and Maoist China, neither of which could have said to been optimal for human flourishing. I'm not saying AI safety has lead to anything this bad yet, but when you start talking about nuclear war being acceptable to achieve your aims, or that we need a world government to nationalize the means of compute because the alternatives are so much worse, it does make me concerned.
More options
Context Copy link