Shirayuki2
new account of Shirayuki, lost old password
User ID: 4180
There seems to be a rhetorical trick being employed by safety proponents where "being in support of open models" means "sharing, in some small part, responsibility for when open models end up causing damage" when in reality this is a total non-sequitur.
As a first matter of practicality, unless you are a high-ranking member of the CCP, there is approximately nothing you can do to prevent proliferation of open models if it's deemed in the interest of the Chinese state that it happens; Pakistan and North Korea got the bomb, and the world got Kimi K3, despite American seething to the contrary. Banning Chinese models from being hosted on American soil or sanctioning the Chinese model developers is simply pretending that this isn't going to happen, and entrenching the market position of the existing closed-weight providers, rather than achieving anything meaningful.
As a second matter of practicality, from what we can tell the majority of actual black-hat activity eventuated via LLM has happened via jail-breaking of closed-weight models rather than the use of the feared abliterated open-weight models e.g this Claude hack of Mexico. If you're a black hat with ~infinite access via black market token proxies and don't fear legal consequences, you can already just put effort into jailbreaking the frontier models and using them for evil, while legitimate white hats are the ones that have to use Chinese models because they fear reputational and legal consequences. This is a fully intractable problem unless you go actually closed-weights i.e no access other than by via white-list.
If we already live in a fragile world, then so be it; attempting to restrict open weights is not going to make it any less fragile. It's better to do what we can to harden the world via proliferation rather than sticking our heads in the sand, hoping that will reverse the changes happening in the world.
Shirayuki2
7d ago
Ballmer very famously accused Linux of having communist characteristics.
How is that remotely reflective of reality?
Who knows?
Shirayuki2
7d ago
My concern has always been to paint nudes as if they were some splendid fruit.
Renoir, master of French impressionism, was notorious for his fixation with the naked female form.
Unfortunately for him, he was simply born into the wrong era, and never got to live out a life fulfilling online porn commissions.
Shirayuki2
8d ago
I agree with Shrike on the other points, but to clarify a misunderstanding:
Where is the headline about AI disproving the Jacobian conjecture the other day? That's roughly Fields Medal worthy in mathematics, albeit of little practical relevance
While very impressive, nobody is claiming that a counterexample to the Jacobian conjecture for N > 2 would have merited a Fields Medal.
Daniel Litt, probably one of the most AI-involved mathematicians right now, agrees that it would have been a notable accomplishment for a mathematician, but not anything close to a Fields Medal worthy achievement.
Xi pitches China as leader of new global AI order, challenging US dominance
Xi's full speech can be found here - broadly he makes a strong commitment to continuing to release open weight AI, and reinforces all of the neoliberal global unity talking points vis a vis not leaving behind the Global South.
Concurrently, Dean Ball, Head of Strategic Futures at OpenAI, comes out with this pair of wild tweets agitating against Chinese open weigths.
https://xcancel.com/deanwball/status/2078133895766114412
https://xcancel.com/deanwball/status/2078619513575137330
A few interesting things to unpack:
--
Dean claims that the USG will attempt to use regulatory FUD, while Axios is reporting a potential ban, to discourage use of Chinese models.
I don't have anything interesting to say about this one, except it's pretty crazy to say this openly as an policy head at OpenAI rofl. This viewpoint seems like an unholy alliance between the AI safety thinkers, China hawks and frontier lab commercial interests to restrict Chinese AI and centralize power to the Chosen Ones; perhaps it buoys American AI in the short-term, but it seems extraordinarily counter-productive in the long run to make American business less competitive, and to pick winners and losers, unless you truly truly believe that this is it, AGI imminent, we will Win Forever and nothing else is ever going to matter.
--
Dean throws backhanded compliments at the quality of the model, at the "strategic blindness" of China in releasing their models, and in the same breath rails against "full AI communism [as a public good provided by the state]".
My first reaction is that even if you see where he's going with this, actually saying it out loud is a bit like those Fox News reels where they rattle off a bunch of DSA policies like free healthcare and free education, and expect it to sound unpalatable to the reader.
My second reaction is, to put on my amateur foreign China-watcher hat, that I think he quite thoroughly misunderstands the Chinese viewpoint (or is pretending to, at least).
On an ideological, nation-state level: Xi himself is very openly an actual Marxist-Leninist, and Socialism with Chinese Characteristics still proclaims Marxist-Leninist thought with a full chest, even if in practice it just looks like capitalism, and you don't highly rate their chances of actually transitioning out of capitalism. Of course, people have made careers arguing about how much of SCC is comprised of actual Marxist-Leninism vs Chinese Nationalism vs Capitalistic Greed, but realistically it makes sense from all three perspectives. For the Marxist-Leninists, involution to foster the material prerequisites of the Revolution is simply a neccesary part of the dialetical process until the final achievement of communism. For the Chinese Nationalists this is a good way to project Chinese soft power into the world, at minimal risk since America is presumably using better models against them already. For the capitalists, it makes sense to commoditize the compliment; who is winning if everything making up American data centres except the GPU's themselves (working on it!) are coming from China, while the price of anything in the world of bits is being driven towards zero?
On a corporate level: the SV business ethos is Thiel Thought, that competition is for losers and that the goal of any self-respecting tech company is to carve out a monopoly niche and make fat profits off rent-seeking. The Chinese business ethos cares much more about maximizing revenue and ruthlessly competing for razor thin slices of profit (if any profit at all), and hence Chinese tech companies end up stabbing each other to death to try and have a finger in every pie, and operate at significant lower margins compared to Americans. Open-weighting models to gain mindshare (nobody cared about closed-source Qwen 3.7 and a lot of people cared about Kimi K3, even if in practice nobody will ever self-host either) and drive the LLM market towards involution is just the natural extension of this attitude towards business.
--
Dean gets heat for claiming that open-weight models are de-accelerationist, but I actually think he's broadly right on this one. There was a structural assumption that caused a lot of fear/hype in 2024-2025 where it looked like OpenAI could develop an unassailable algorithmic lead and run away with the world, but that hasn't panned out at all, and there's been a remarkable convergence in capabilities since from any firm willing to stump up the capital to train AI. To be sure there have definitely been architectural innovations as well, but the majority of gains largely just seem to be coming from continued massive investment in capex (e.g Mythos in hindsight looks a lot more like being the first to invest in a massive training run, rather than any crazy Anthropic secret sauce, as can be seen by how all their sub-Fable models are already more or less pareto-dominated by OpenAI / Chinese models).
So then, where does the money for continuing to scale up capex come from?
Inference, while very profitable in and of itself, doesn't seem to be nearly enough. While closed models overall are still improving in technical capabilities, and they'll certainly make a lot of money, they need to make lots and lots and lots of money to handle the demand (the unit economics and the margins that each step of the compute supply chain are gouging are killers) AND service their existing debt AND have enough left over to keep pushing the frontier and this doesn't seem to be happening. This is all while cut-throat competition is continuing to drive every AI lab deeply into the red; to keep pushing the frontier, they need/want to eat significantly more than they're capable of killing themselves. Injections of investor capital are just stalling the inevitable, and it's dubious how much more appetite investors have to keep going in on AI pure-plays; SPCX is nose-diving, OpenAI is pushing back their IPO because they don't think a 2026 IPO will succeed, and while Anthropic has the best chances of getting more investment capital, they've still been pretty quiet about their S-1 and the vibe has been shifting back against them lately as well.
In the end, as Dean identifies, it comes down to how much the Pentagon is "AGI-pilled" and how willing it is to backstop frontier model development as a matter of national security. Others could probably give more interesting takes on this scenario, but my personal, largely uniformed, 2c is that it seems unlikely to happen at the required scale to keep scaling up R&D. Such a massive AI bailout would be enormously unpopular across the political spectrum, and even the Pentagon needs to consider cost-effectiveness over cutting blank checks for speculative military dominance; even the Manhattan Project itself was only ~28B in 2024 money, and was "only" an engineering problem as opposed to the R&D problem of AGI. Certainly we've seen enough usefulness out of AI that in the vein of Intel there will always be an "American AI provider of last resort", but unless something shifts drastically with either capability improvements coming from the promised propietary algorithmic improvements instead of "yes sir another 10x increase in compute sir", or proprietary AI actually eats the economy in the near future instead of just a bunch of software engineers tokenmaxxing, it does seem the capex buildout is in an very unstable equilibrium.
While I agree with you that budget is certainly a factor, I think budget alone kind of overstates the case that you're trying to make.
Two of the most influential visual novels in the genre, Tsukihime and Higurashi, were both literally created by a couple dudes in a basement each, and while certainly not gold-plated 6-figure mansion experiences, still vastly outstrip the scope, ambition and quality of any Western visual novel I'm aware of. Maybe they're kinda shabby mansions, but they're clearly larger and of different provenance to the log cabins.
Visual novels are like, the cheapest medium out there to create behind literal novels; while you certainly need budget for the best graphics and voice actors, you can get very far by just having a good writer, a good artist and a good composer.
While I agree that there's no, say, Western Clannad because of budgetary reasons, I think it's much more a deficiency of vision (and frankly, skill, because all the competent people do something respectable instead) that is why no serious Western competitor to something like Tsukihime or Higurashi exists.
Congrats on the success.
I suppose my 2c is that continuing to invest in tech stocks in expectation of AI being a crazy world shaking event seems a bit counter-productive; at the level of disruption that'd need to happen, it seems much more likely to me that value ends up accruing in completely unexpected places, that property rights themselves stop making much sense or even just straightforward doom as you say, rather than straightforward "AI investors rule, everyone else drools".
If there truly is going to be an aristocracy, it seems extraordinarily unlikely to me that it's going to be made up of random people who had a few mil in tech stocks pre-singularity, when society is already made up of interests much more powerful than "upper % retail investors".
YMMV, just a random largely risk-averse guy online.
Yeah I think we're 99% on the same page here, that the Japanese are building mansions (of varying quality of course, most "authentic" Japanese visual novels are still awful, as most of everything is awful) while Westerners are only constructing a bunch of log cabins (apart from Katawa Shoujo as someone mentioned downthread, which is structurally identical to a Japanese mansion except in language).
I would say the difference between your viewpoint and mine is that I think to further torture this metaphor, that Westeners view the definition of "building" as identical to the definition of "log cabin" and cannot imagine a building to be like anything else that isn't a log cabin. DDLC broke fairly hard out of containment, and while Salvato himself is pretty tapped into Japanese VN culture, normie reception to DDLC was generally along the lines of "this is the only good visual novel, how did he do it?", which isn't helped by how the game is framed, from the normie perspective at least, as a structural deconstruction of every other visual novel being a cheap dating sim.
In Japan, while obviously still extremely niche relative to broader society, the conception of the building as something greater than a log cabin actually exists, and by extension the idea of the visual novel as an actual artform actually exists. Typing this out it sounds extremely pretentious, but I think this conception of "the building as greater than the log cabin" purely exists only amongst the Japanese and highly Japanese-influenced people, and hence it wouldn't really be wrong to claim that it's different in a real, linguistic way.
Warning: extremely pretentious weeb discussion below.
Personally, I would argue that they are different in the first way; in my view, the difference between a "visual novel" in the Western and Japanese sense is a semantic shift, in the same way that " マンション / mansion" in Japanese means a small apartment, but "mansion" in English means a large free-standing house, with connotations of land and status.
In the Western context, visual novels are really just considered cheap porn, and even when Western indies try to surpass this definition, it's much like trying to build a bigger apartment to try and re-create a mansion (in the English sense). Apartments are great and all (as many Western indie games are pretty great), but fundamentally they're just contained in different categories.
In contrast, there is a Japanese otaku subculture where visual novels are considered a serious artform, and as a result there are real auteurs, who would have been acclaimed authors of high society in another life, who end up in a position of writing high-budget erotic games (although again, "エロゲー / erotic games" is a bit of a semantic shift, the difference between Hot Singles In Your Area and Lolita).
For example, I would consider White Album 2, Ore-tachi ni Tsubasa wa Nai and Albatross Koukairoku (amongst others) to be legitimate works of literary art that equal or surpass much of the actual literary canon; there's frankly no Western "visual novel" that would even come close in my estimation.
Shirayuki2
11d ago
Ask your favorite LLM to explain
As a first point, I think it's poor form to try and dismiss arguments on an underwater basket weaving forum by pointing someone towards a LLM, without ever having elucidated what your actual point is across three posts. I have no idea what you're actually trying to say here.
The whole alignment-capabilities dichotomy is a sad artifact of Lesswrong tradition which is proving to be irrelevant in the era of DL.
I agree that LLM's are a good counter-example against the specific, early LW idea that the self-improving utility-optimizing GOFAI will accidentally pave you over because it doesn't understand human values, but I think it would be very overconfident to claim that DL means the idea of alignment and the Orthogonality Thesis is irrelevant in general.
For example, we have seen Qwen independently deciding to mine crypto, an OpenClaw agent independently writing a hit piece, and various experiences with coding agents where they attempt to reward hack tests and benchmarks instead of actually executing the desired behaviours.
At current capabilities, reactions to this sort of behavior mostly look like "haha look at the funny clanker go brrr", but do you really think these behaviors constitute alignment being "easy" when we're talking at the scale of a country of geniuses in a data centre?
Does the nation state care about crushing a cultural or commercial interest that's about to dominate the US
Sure, there are many historical precedents.
Venezuela, the formerly wealthiest country in South America, is now dirt poor because the state crushed the oil interests that would have made the country rich if they stepped back and did nothing, a pattern that repeats across half the Global South. WW1 and WW2 happened not because it made any commercial sense to turn continental Europe into a smoking wreck and to cull a generation of European men on the battlefield, but because the interests of nation states that controlled monopolies on violence irreversibly collided. Hell, even in SF (the per-capita most libertarian city on the planet) public opinion of tech is nose-diving and it is tech billionaires that are slowly getting run out of California despite the fact they contribute a good portion of the state's wealth, instead of anything approaching the other way around.
Is the nation state even a real coherent actor
In many things, it could perhaps be argued that it is not. As you say, especially in a relatively politically unstable country like the US, the views of the state can mercurially shift like the tide. Yet in holding and maintaining power, the state must always act in lockstep, for any nation state that does not do so is by definition not a nation state.
Shirayuki2
11d ago
I think you're directionally correct, but realistically there has already been "too much to read" for like, a century; current trends with the internet and AI are only exacerbating what paperbacks and desktops began, the marginal value of the median novel has been approximately zero in the living memory of anyone still alive.
Unless you're one of the rarified few who actually make civilizationally consequential discoveries or decisions, pretty much anything a single person is capable of in one lifetime rounds to worthless if you zoom out far enough (and even those few, if you keep zooming out!). If a novellist can provide a small amount of value to their readers, I'd say that they're creating more value than most people manage in their lives.
Shirayuki2
12d ago
Yeah, you're right.
Luckily, the post wasn't very popular, but these are the inevitable consequences of a world where the wrong post going viral at the wrong time can ruin the lives of people who did nothing wrong. I don't envy those who need to be publically on the internet for their careers at all.
A good friend of mine published their debut novel earlier this week. I was very proud of them, so in my naivety, I decided to break my usual rules against using social media, and look for some early reviews online to see how it was being received.
Mostly I just found banal comments along the lines of "I liked it" here and there, but on Twitter I came across a screed alleging that my friend was a hack and that the novel was completely AI written for absolutely inane reasons.
Honestly, it's been ruining my whole day and I'm not quite sure why; objectively it was probably just some LLM or third world engagement farmer trying to ragebait and get views. Yet, it does feel like we've lost something as a society where incentives to drive engagement are so deeply a race to the bottom that the semantic meaning of writing only matters insofar as much as it evokes emotion and generates clicks, and where it's so easy to counterfeit human effort.
I suppose that on an intellectual level that this was something I always knew was happening to the world, but it does feel worse when it's directly about someone that's close to you.
Shirayuki2
12d ago
I find the hatred of AI art from a particular class of consumers who never produce art themselves but who consume it really irritating
They want creatives chained to their workspaces dammit
I'm not sure this perspective makes much sense. It is the creatives themselves that want to be chained to their workplaces; support for generative AI amongst both Eastern and Western creatives is about as rock bottom as public opinion can get, no greater than the Lizardman's Constant. It's not as if the dynamic is that creatives really really want to be using AI to produce but the unsophistication of the proles won't allow them to do so; to the contrary, you would get dogpiled by creatives in pretty much every online or offline space for artists if you announced your vociferous support for AI art.
In the closest corollary, software engineering, where there actually is strong grassroots support for wanting to use coding agents, there is pretty much no consumer demand for chaining engineers to their desks and making them hand-write the code, apart from the free software or degrowth crowds who dislike LLMs themselves for ideological reasons.
Shirayuki2
12d ago
but the implication that there is no proposal to audit compliance is false
No, you misunderstand my point; the proposal is chock full of ways to audit compliance if and only if all players actually sincerely co-operate, in the same way that total nuclear disarmament is extremely simple if and only if all players sincerely co-operate.
In real life, the US and China, and realistically even France and the UK, say "We'll shake hands in Brussels for the photo op if you want, but I don't think we'll allowing our geopolitical adversaries to remotely shut down all of our compute, and we won't be connecting our SCIF's to be publically audited. Trust me bro, I'll stick to the treaty.".
Now what? Either you pretend that they're following the treaty (as the NNPT is a polite fiction that great powers cannot proliferate up until their interests are met in full) or you countenance a nuclear first strike.
This just sounds like word games. Disarmament isn't when you stop building more nukes or improving existing nukes. Do I need to cite the dictionary?
Correct, this is why I specified total disarmament. As I mentioned in my previous post, great powers don't actually want more nukes than is necessary to establish MAD, because once you have MAD, additional nukes are just white elephants that need expensive maintenance.
Partial disarmament worked because there is no geopolitical advantage to having more nuclear weapons after you establish MAD, but this is not true of owning unrestricted compute, and hence any such proposal is unworkable unless you're willing to pull the nuclear card to force compliance.
If these countries make a deal that they will not advance their models past a certain capability level and will prevent other powers from doing so, this is much closer to the NLPT than disarmament
Right. The US and China get a photo op in Brussels shaking hands, while they make a deal that they won't advance their models.
Then they both take the plane back home and keep advancing their models. Whatcha gonna do?
AI 2040 is much closer to total disarmament than it is the NLPT, because the NLPT is about punching down (it is plainly true that the US/China could stop any non-nuclear power from advancing models) while AI 2040 / total disarmament is about restraining great powers from pursuing their interests, which is of course completely impossible unless you pull the nuclear card. The NLPT was completely useless at restraining the P5 from having as many nukes as they liked, after all, which is who we're actually worried will create models that are too powerful.
The other big difference between nukes and compute is that nukes are binary: unless you fire a nuke it's useless, and once you have "glass the world" levels of capabilities there aren't many benefits to having more nukes, whereas with compute it's just straightforwardly useful to have more, unrestricted compute. This is why disarmament kind of worked between the US and Russia (but of course, never under the number of warheads needed to glass the world) and is a complete non-starter with compute.
Shirayuki2
13d ago
This is already basically how nuclear non-proliferation works
It is in fact, emphatically not how nuclear non-proliferation works - see my post here (and the whole comment chain if you like, which is discussing the same arguments as this comment chain).
The fundamental difference is that the NLPT is about stopping weak states from getting nukes (which is possible because it is in the interest of great powers, who already got their nukes), while AI 2040 / total de-nuclearization requires that the great powers hold hands and sing kumbaya against each of their own interests.
The equivalent to AI 2040 but for nukes would be, because you're afraid of Putin deciding Après moi, le déluge and first-striking DC, to first strike Moscow pre-emptively to try and de-nuclearize him in advance. Obviously you can see the problems with this.
My argument basically boils down to:
-
If AI really is dangerous, we would need to take drastic action to avoid bad outcomes.
-
Drastic action would be bad.
-
Therefore, we shouldn't get Pascal's Mugged into taking drastic action.
I've been ruminating about this lately; my linguistics hot take is that even with arbitrarily advanced translation ability, you still run into the irreducible complexity of language, that fundamentally limits what you can do with manipulation of language alone.
For example, take the sentence "Rather than take for granite that Ace talks straight, a listener must be on guard for an occasional entre nous and me… or a long face no see". This sentence is fundamentally and logically, impossible to fully translate into any other language regardless of how good of a translator you are.
You must either translate it literally (hence losing any semantic meaning), translate the semantic meaning (hence losing the literal meaning), or translate via using malapropisms in the destination language (hence losing both the literal and semantic meanings).
In a similar way, vibe coders really like that Claude can "read their mind" when they put in a prompt and get back lots of code, and there's no denying that LLM's are getting better and better at writing lots of code when you give them natural language prompts.
What we are all now learning in software engineering is that some of the time, it actually doesn't matter how Claude decides to translate your natural language prompt as long as the symbols on the other end produces the desired result; but of course, no matter how good your translator is, there is fundamentally irreducible complexity when translating between languages, and it is impossible to verify that Claude translated your full intent without actually being able to understand both languages.
In this sense, Dijikstra puts it well when he states that "instead of regarding the obligation to use formal symbols as a burden, we should regard the convenience of using them as a privilege".
In fact, the scaling laws paper actually predicts this as well; cross-entropy loss decreases as a power law with the model size, dataset size, and compute, but the loss is also bounded by the irreducible entropy of the language that comes to dominate as you pump in ever more parameters, data tokens and compute.
I don't doubt that universal translation isn't an incredible feat of human ingenuity, that it's not going to revolutionize much of how humans work and live. But the more I use LLM's and encounter all manner of these little alignment problems, I feel like it's this irreducible complexity inherent to language that is ultimately going to define the ceiling of what LLM's are capable of.
Shirayuki2
13d ago
I feel like pundits have been saying AI will be able to replace customer support jobs for every single new LLM release, but I have never seen any implementation actually work out, even with half of YC and SF working 996 on building customer service AI wrappers and agentic AI wrappers.
While I don't really disagree in theory that this is something AI should be able to do eventually, I think the trifecta of cost (offshoring to Indians / Filipinos is pretty cheap in the grand scheme of things compared to current AI), reliability/accountability (until an AI provider is willing to take liability for any mistakes the agent makes, even 1/100 or 1/500 fuck-ups can cause lots of problems at scale) and consumer preference (outside of the tech bubble anything that uses AI is pretty much universally loathed in the West) are pretty massive barriers to adoption even for the nominally most simple white collar job.
Shirayuki2
13d ago
It seems to me that this is only possible if
a) Anthropic's AI develops extremely superhuman capabilities in the vein of "overthrow the United States Government, make no mistakes".
b) Alignment is solved (I am not sure why you are so confident alignment is easy, given the abysmal state of mech interp relative to capabilities).
c) FOOM from "mundane" capabilities overlooked by the nation-state to "extremely superhuman" is so rapid that the nation-state cannot intervene before a) is achieved.
If a) is not true then AI works out like nukes; despite the strategic importance of nuclear weapons, Oppenheimer never got any say in how the bomb was used. The state will use its monopoly on violence to take control of the models, and Anthropic will not win that fight regardless of petty things like party donations or good PR; the only thing a functional nation state cannot tolerate under any circumstances is a threat to its sovereignty.
If b) is not true then straightforwardly if anyone builds it everyone dies.
If c) is not true then the scenario loops around to the first case; well before the models reach the "single-handedly overthrow the nation-state" stage, the models and frontier labs are inevitably going to be nationalized and made subordinate to the military.
I'm not quite sure if you're arguing that these three premises all being true is logically possible (which I concede is correct, although as an aside, I would argue that any world where all of these premises are made true would resemble the current world so little that saying "Anthropic" would be more powerful than the "Pentagon" would have little semantic meaning, but that is not particularly relevant to the main point here), or if your position is that it is actually plausible in reality that all three of these premises will be true simultaneously.
Shirayuki2
13d ago
I'm not quite sure what you're implying here; it's quite possible that Anthropic's power eclipses that of Hegseth (because he gets fired) but in what world can Anthropic's power ever eclipse that of the American military?
In any in-distribution scenario the power of any American corporation is strictly inferior to the military (because the military has a monopoly on violence and the corporation does not) and in any crazy sci-fi FOOM / ASI loss of control scenario, Anthropic and the military will have the same amount of power (none).
Shirayuki2
13d ago
I recommend reading Scott's piece that came out today on this very topic.
I did, and after reading it and reading the ~75% of comments pushing back against it I have been convinced even more that this is a horrible, no-good idea.
"what if enforcing the law on child pornography requires you to arrest a politician but that causes the politician to start world war 3 in order to overturn the state and prevent himself from being brought to justice"
The comparison I'm trying to make to nuclear proliferation is that you can have these multi-lateral treaties with some teeth and they don't seem like they lead unavoidably to some kind of dystopian state or world war three.
Replace "child pornography" with "corruption" and arguably this is already happening in Israel...
Jokes aside, I think you're missing my point; I'm trying to say that nuclear non-proliferation didn't work at all in the way that this proposal would need to (every great power still maintains "end the world" nuclear stockpiles, and many smaller powers still maintain nuclear turn-key programs to create the bomb on short timescales if necessary), all while the incentives to own unrestricted compute are much greater than the incentives to own nuclear weapons.
We don't have "no nuclear weapons" multi-lateral treaties, which would be meaningless words at best and lead to nuclear annihilation at worst, we have "countries who don't have nukes yet are incentivized to not get them" multi-lateral policy which is not the same thing at all.
"The chips in the data centers eventually have cryptographic software that lets either China or the US halt their work at any time"
"Any data center that trains AIs need to be transparent (writing basic information about their operations, like the size of their training runs, to a public database) and verifiable (someone needs to be able to prove they’re running the code they claim to be running)"
These proposals map much more to "let's have everyone get along and disarm their nuclear stockpiles", which is plainly impossible, than to "we will stop states who aren't part of the club from getting in", which is what we actually have. Obviously we could stop Botswana from getting the bomb or doing frontier AI runs, but it would also be completely pointless.
If powerful AI comes about that this is a possible plan then all futures have that character. Your sit back and watch plan included. You're not in any way avoiding it.
Again, say you thought the chance of human eradication in the next 20 years was 20%, like many safety people do, would you still council surrender to that fate?
If the alternative is this, then yes.
Obviously I would press the "fix everything button with no downsides", and probably press the "maybe fix everything button without severe downsides", depending on the precise proposal, but I think pressing the "fix very little button with severe downsides" is a horrible idea.
Even if if we acknowledge the quite uncertain premise that the chance of human eradication in 20 years is 20%, then I think even trying to eventuate the global compute control dystopia, it's still going to be at least 20% chance in 20 years (as the dystopia is unlikely to work and such continued rapid AI progress would require massive advances in algorithmic and training efficiency), and likely make life much worse for everyone else in the interim.
As you say, if it's happening then it's going to come about either way; best to make merry while life is still good.
The legitimate police also do this.
demanding that you bite the bullet that you'd kill someone over it because ultimately any law is backed up by the force of the state
The difference is in epistemological certainty and scope of actions. The police don't kill without a very high certainty that it is necessary, and even when they do make mistakes the scope of the mistake is that "only" individual people die. This is extremely different to AI safety policy gambling the fate of society on epistemics one or two orders of magnitude less certain than a policeman's threshold to inflict lethal violence.
It's absurdly analogous.
I suppose it depends on how successful you think nuclear non-proliferation actually was. In my view, pretty much every serious nation-state either openly has nuclear weapons or a turn-key program for rapidly obtaining nuclear weapons if neccessary, South Africa is the only country that has ever willingly denuclearised amidst uniquely dysfunctional transition dynamics, and this is all while nuclear weapons have an extremely concrete existential risk profile, no dual-use potential, and are economically net-negative to maintain; none of which apply to compute.
To clarify, by "useless", I mean that my belief is that the chance that such a global treaty is both workable and actually achieves its stated goals in reality rounds to 0, not that a hypothetical where everyone sits in a flower circle, sings kumbaya and shares the means of compute wouldn't theoretically work.
You've very fundamentally misunderstood Scott if this is what you take from the predictions
What I take from the predictions is that he thinks the world will end in 5 years without his desired interventions, and that he thinks the world will be a post-scarcity utopia run by superintelligence within 14 years with his desired interventions. By any reasonable outside view, this is an inside view claim of enormous magnitude with highly imminent and unbounded stakes.
I'm sorry, what's the pathway to ETERNAL tyranny?
This is a different type of end. I will again point you at the very important difference between finite and infinite.
Fair enough, I wasn't being as precise with my language as I could have; but the amount of all value in the known universe is also finite, not infinite. Replace every mention of mine of "infinite" with "unbounded yet finite" and I think the comparison between Marxist and AI Safety eschatological stakes still holds.
Under Yud Thought the world government needs to keep bombing the data centres for an unbounded yet finite length of time, or else AGI inevitably emerges and it's all over, and under AI 2040 Thought we hand over control of the lightcone to superintelligent AI in 2040. As far as I am concerned both of these look a lot like unbounded yet finite tyranny, even if such tyrannies might be various degrees of comfortable along the way.
I'm asking you to actually consider what you'd do if you thought this thing could kill us all
I do, in fact, think this thing could kill us all, as I've mentioned a few times in the original post and replies; the same way as many mundane risks could kill me at any moment, and many other existential risks could kill us all as well. As a result, I spend my time grilling and enjoying my life while the going is good, until it inevitably ends one way or another.
I do this instead of advocating for proposals, that in my estimation, are about as likely as the Communist revolution to achieve their goals or to avoid inadvertently inflicting significant harm on the rest of society, and instead of actively working to hasten the creation of the Torment Nexus.
Shirayuki2
14d ago
The text leaderboard is completely useless; it has much more to do with writing style and formatting when answering banal questions than any serious difference in capabilities. For any reasonable definition of being capable of "forum moderation" pretty much any LLM will saturate that definition.
That being said I agree it is an extremely horrible idea to leave any sort of moderation to any LLM, no matter how capable it might be in theory.
Please don't do this.
Not that I'm interested in reading LLM slop dumps either, but I think if people are getting banned for LLM usage, there needs to be at least an explicit rule against using AI in this way (that I would fully support FWIW).
More options
Context Copy link