Shirayuki2
new account of Shirayuki, lost old password
User ID: 4180
The chance of it killing us all is coming either way
I generally agree, in the sense that "WW2 was inevitable as a result of complex geopolitical factors post-WW1", that "if the fate of human civilization is to create the Torment Nexus, then the Torment Nexus will inevitably end up being created", but obviously that didn't clear Hitler's name and being on the vanguard of potentially creating the Torment Nexus is simply not going to make AI labs popular.
I mean of course they haven't because those dollars haven't even been meaningfully spent yet because we haven't built the datacenters
I mean, is that supposed to make it a more appealing thesis for the average person? "You should give us a trillion dollars to build out datacentres that aren't being built, and maybe if you give us enough and we finally succeed you'll be lucky to get something out of it and avoid the permanent underclass or being drone striked by Skynet" is not exactly stirring rhetoric. I'm not saying that AI is useless or that it couldn't theoretically generate a lot of consumer surplus, but that's clearly not what the public messaging on the topic in the West is pushing.
The median person who hates AI thinks a literal bottle of water is obliterated from the universe every time you query chat gpt for anything
While it's true that populist anti-AI arguments are not very good and mostly arguments-as-soldiers, that's distinct from not having good reasons to act that way in the first place. There's a dedicated core of people who hate phones, cars or fracking, but most people just enjoy the benefits to their lives from those technologies and move on with their lives; there isn't really such consumer surplus and many externalities from the current state of AI, so it's no wonder why people start turning to populists and arguments as soldiers.
I think there's a fair chance this whole thing ends up with all of us dead
as well as a fair chance that progress peters
Are you really surprised why a technology where its proponents and developers are saying that it has a solid chance of killing everyone and/or consigning the median person into permanent unemployment at best, or creating a titanic financial bubble that's going to bring everyone's retirement with it if it doesn't pay off at worst, is extremely unpopular?
As a software engineer I enjoy using coding agents and I do think with demographics and fertility being what they are, the only real choice as a society is leaning into AI and automation as much as possible even with the commensurate risks, but frankly the median person has really not seen proportional consumer surplus from the trillions invested into post GPT-4 or so LLM development, and in return all they get is dealing with a tsunami of online slop, vibe-coded software that gets worse even as engineers boast about how productive they are, and higher prices for power and RAM.
At least in China they have the message discipline to tell people that AI is going to be used to improve people's quality of life, but all SF can do is jerk off about the AI-induced permanent underclass happening any month now and how dangerous AI is is going to be; it's obvious why the median person hates the AI build-out.
Shirayuki2
4mo ago
While I agree with the premise that you hardly need to work 40 hours a week for 40 years to maintain at least equal and in most respects significantly superior material conditions than someone from the 1920s, it's impossible to get a 1920's level of existence via welfare or FIRE - living a 1920's life alongside friends, family and potential romantic partners in 1920 is significantly different to voluntarily choosing such a life while all your peers are living in 2026.
In that respect it's hardly the fault of the system, capitalism, or wealth inequality though, but simply the natural consequences of choosing to live like an oddball; at the end of the day most people complain, but their revealed preference is that they'd rather work more to consume more and fit in.
Shirayuki2
4mo ago
While I align with the idea that in the limit the only real possible outcomes of superhuman general AI are doom or utopic abundance, I agree that the third point is proving too much. Possibility of AI being a bubble aside, even granting that AI lives up to the hype and drives unprecedented productivity, it seems extraordinarily difficult to determine who actually ends up capturing the value from that productivity.
Nvidia is a software company in a trenchcoat vulnerable to having their moat and margins forded by sufficiently good SWE AI, FAAMG still derive a supermajority of their revenue from vulnerable middle-class consumers directly and indirectly, and the American frontier labs are potentially vulnerable to having their margins on inference undercut by Chinese lab competition and tokens turning into a commodity.
Personally, my thesis is simply to be largely all in on global total market indices - reasonably diversified against a potential bubble, while insofar as property rights are preserved and AI creates unprecedented value, it gives me the best chance to invest in the someone primed to capture that value, and presumably even the "modest" returns on an index fund are going to be good enough if AI does end up being that transformative.
Of course, in the more exotic doom / utopia scenarios one's investment mix is not going to matter very much either way; betting on the rapture has never been a winning strategy.
Realistically, I think the only solutions that work at scale are technological advancements that obsolete the need to exploit the commons.
There's literally never been an instance where people "just" decide to collectively stop defecting, rather the payoff matrix gets changed by technological advances and it becomes irrational to defect.
Limiting CFC's went so well in large part because it ended up being pretty easy to transition from CFC's to HFC's, and peak fossil fuel is approaching not because of the environmentalist or degrowth lobbies but because the costs of renewables are falling precipitously. On the other hand, trying to prevent antibiotic resistance has pretty much gone nowhere because antibiotics are so hard to replace - I have much more faith that we'll get better ways to kill bacteria before any progress is made trying to coordinate usage at a global scale.
I think it just depends on how adversarial the judging ends up being rather than saying anything about capabilities.
If the judges have to stay within guardrails then 3.5 could probably win the longbet, but if they're allowed to exploit jailbreaks or known LLM failure cases, then nothing short of ASI is going to pass the test.
Shirayuki2
5mo ago
This is much more maximalist than even the AI 2027 crowd or the actual frontier labs themselves but I respect providing specific events and an end date on the prediction.
My own expectation is that none of this happens by the end of 2027 except a tranche of models that are notably better at RLVR'd tasks and the Copilot button (shorthand for AI getting more integrated into white-collar workflows). Let's see what happens in 18 months.
Shirayuki2
5mo ago
their R&D spend is worth it, even if crazy
Well, burning other people's money to try and build a moat is obviously worth it for the frontier labs. It's yet to be seen whether that spending will be worth it in the sense of paying off investors or building the labs a durable lead, or whether the models will end up commoditized and value accruing elsewhere in the stack.
Shirayuki2
5mo ago
While inference having high margins is true, there are two things to keep in mind here:
Amodei has never said that models are actually profitable on a per-model basis, only that they hypothetically could be. While this might be true, there are trillions of dollars on the line to insinuate that it's true, and personally I wouldn't trust any rumors about financials from a private company who can massage them however they please.
Spending the GDP of a small country on R&D on the promise of getting a commanding lead is why OpenAI and Anthropic have trillion dollar valuations to begin with. There's no such thing as a frontier lab who can cut their exorbitant capex and coast on the margins from inference, as that's a one way road to getting cut-throat commoditized.
I feel like a lot of people in these replies are talking past each other.
My 2 cents:
Are LLM's useful tooling for finding vulnerabilities for security researchers?
Yes, I think this is undeniable at this point; LLM's are exceptional at uncovering software flaws, bugs, and vulnerabilities, and are going to significantly change how cybersecurity is practiced, as can already been seen by how vulnerability disclosures have recently quantitatively spiked like crazy.
Is Mythos better than the other available models at finding and exploiting vulnerabilities?
Yes, Mythos really being a stronger model for cybersecurity applications is almost certainly the case: this XBOW report is a good read on its capabilities.
Is Mythos a super-hacker that's going to break cybersecurity for good?
No, this seems unlikely and driven by good marketing from Anthropic and online hype. Mythos isn't making the Move 37 for cybersecurity or discovering vulnerabilities beyond human comprehension, it's just an iterative improvement over the current tooling combined with a lot more compute and attention suddenly being used to uncover security vulnerabilities. I suspect that the same amount of compute, security researcher attention and buy-in for Project Glasswing applied to the previous generation of frontier models would have uncovered the majority of security issues that Mythos did.
It's also worth noting that there are apparently 11 Curl CVE's in the current release cycle, where the new CVE's did not use Mythos, which seems to disprove the idea that Mythos was not all that effective on Curl because it was uniquely hardened or secure.
Should LLM's being good at vulnerability discovery and theorem proving be an update on LLM's eventually reaching AGI?
YMMV, but to me, the recent headline mathematics and cybersecurity achievements haven't really changed my view that AGI emerging from LLM's seems unlikely. From an outsider's perspective, most of the recent gains in model performance look to have come from RLVR on coding, math and cyber. While very effective at improving performance on those tasks, it seems that RL has largely failed to further generalize intelligence beyond the specific RL'd areas, and if you look at SimpleBench or the AI RP community, seemed to have regressed performance in other areas of intelligence.
I think it's telling that all of the achievements of LLM's being held up over the past ~18 months (METR eval, CCC compiler, theorem proving, cybersecurity), while extremely powerful and which make me bullish on the utility of LLM's, are all tasks limited by requiring an external oracle for verification, and where there's no penalty for failing during intermediate steps. I personally think it's quite likely that LLM's eventually become superhuman at proving theorems and exploiting vulnerabilities given sufficient compute, but still cannot manage a restaurant, write an interesting book or autonomously maintain a software project.
Shirayuki2
5mo ago
What are some unusual items you've got on your bucket list?
Japan
Losing the war and being occupied by the allies was probably the best thing that could have happened to Japan's economy, as opposed to dragging them down.
The American occupation broke the Japanese military-industrial complex and forced them into exports and free markets, funneled them large sums of capital via Korean war procurement, American market access and technology transfers, and extended Japan the American security umbrella. The Japanese economic miracle wouldn't have been possible without losing the war and getting dragged into a modern economy by force.
India
While they never went full communist like China, there's a good argument that India could be decades ahead of where they are now without Nehru and Gandhi fucking around with poorly implemented socialism. The over-emphasis on heavy industry, licensing and central planning, failing to implement any real land reforms, and essentially being closed off to trade prior to 1991 were practically the completely opposite conditions as to what made the four Asian tigers so successful.
While I agree that India has different issues to East Asia, having a 50 year disadvantage on Japan and a 10 year disadvantage on China in liberalization did them no favors either.
Shirayuki2
5mo ago
Just wanted to say thanks for posting this.
I've been in a bit of a depressive spiral for the last few days and this post really helped.
Shirayuki2
6mo ago
My understanding of calvinism vs middle knowledge doctrines was that the main difference is that Calvinism treats free will as compatibilist while Molinism treats free will as libertarian.
Under either doctrine God must still have foreknowledge of children being born with genetic defects that kill them a few days out of the womb and foreknowledge of those destined to die during natural disasters; otherwise how could He be omniscient?
Under theism, you can consistently make choices that go against your self-interest and sacrifice both self-interest and prudence for the sake of morality
Sure, but the point I'm trying to make here is that moral epistemology becomes incoherent, if you abrogate omnibenevolence as understood by human moral intuition. If the meaningless suffering that pervades the world is all part of His plan, then there is no reason to morally privilege doing anything over, say, kicking as many puppies as possible. For all I know that kid drowning to death over there is all just part of the plan, and He instead wants me to kick the shit out of more puppies instead of saving the kid.
Maybe skeptical theists can bite that bullet, but it's not a bullet that I feel capable of accepting myself.
Shirayuki2
6mo ago
While I think Buddhism in general has useful lessons in reducing suffering even for an atheist, I don't understand at all how western, secular Buddhism can be logically sound without being a suicide cult.
If you accept dukkha, samudaya and nirodha, but you don't believe in the Right View, in the tenets of karma or rebirth, it seems immediately clear that that instead of the Eighfold Path, the much more efficient method to achieving nirodha is the Singlefold Path of a bullet into your skull.
Shirayuki2
6mo ago
I have to admit I find theodicies appealing to skeptical theism abhorrent, and personally I've never managed to get over the problem of evil.
Why should man do anything good, why should man do anything at all, if human moral intuition is meaningless and the most profane acts can be justified under an unfalsifiable appeal to the greater good?
Rats and ants may not understand the motions of men, but neither should rats and ants have any reason to worship men except for the last argument of kings, the threat of pure brute force and violence.
If God's vision is that children should die screaming in unimaginable pain and that the Ichneumonidae should eat caterpillars from the inside out, then frankly in the footsteps of Ivan Karamazov, I don't want anything offered to me by such a god.
Shirayuki2
6mo ago
US-China great power competition looks a lot less like god-fearing capitalists vs godless communists and a lot more like godless capitalists with Anglo-Hispanic characteristics vs godless capitalists with Chinese characteristics.
America is not god-fearing in any meaningful sense compared to the America of fifty years ago, and China is not communist in any meaningful sense compared to the China of fifty years ago.
Shirayuki2
6mo ago
It isn't really arguable that modern society is failing to provide the former to a far larger portion of the population than it did in the past
I agree, but this doesn't really have anything to do with inequality. Most of South America and Africa have vastly higher Gini indexes and much more blatant, corrupt wealth inequality than any developed country yet retain much higher TFR's, while the social democratic Nordic countries living under the law of Jante have amongst the lowest TFR's worldwide. Being rich, free & educated, having the optionality in life to do anything in life at the expense of having children, social atomization and access to smartphones seem like much more causal factors to plummeting rates of family formation.
At the same time, a lot of the visible concentrations of wealth in modern society are nakedly and undeniably antisocial.
I agree that a lot of the aesthetics of the modern wealthy are off-putting, but as I mentioned earlier, "powerful people act in upsetting ways" is not a solvable problem as long as the fundamental ability to concentrate power through technology exists at all. Nobody remembers the man that Luigi killed and nothing changed whatsoever. If it's not the current crop of people seizing the reins of power to enrich themselves, it'll simply be someone else stepping up in their stead.
Shirayuki2
6mo ago
Furthermore, inequality is a proxy for an uneven distribution of power.
I suppose my view is that wealth isn't power, power is power. Any coalition capable of unseating the billionaire class would by definition hold more power than the current wealthy. I'm not sure it really makes much difference whether it's a Langley spook, Hague bureaucrat, tech billionaire, or CCP party member that holds the reins to ultimate power and status.
As long as technology exists you'll get centralization of power, but as long as centralized technological power doesn't exist you get Haiti or South Sudan, the Hobbesian life in a state of nature.
relative prosperity gates access to some goods that are essential for happiness, like housing and a mate
Extension du domaine de la lutte. The progressive ideal of re-distributing wealth is at least logically possible, but it's fundamentally impossible to re-distribute everyone a big house in the best locations and a high status mate. If being better than others is essential to happiness, then perhaps that is humanity's punishment for eating the forbidden fruit.
Shirayuki2
6mo ago
Sorry to hear that.
Unfortunately there's a lot of this going on in tech and white collar work as a whole, really, where the LLM's really can't do the work, but some executive assumes they can and so people get chopped in anticipation, or where the company is struggling due to macro-economics or just plain bad management and people get chopped using AI as an excuse.
Best of luck with your other work or with starting the new career.
Shirayuki2
6mo ago
I was planning to write up a larger top-level effort-post on this topic, but since you've already made the top-level I'll post the notes I was drafting.
For the last few days, I've been reading about the Sam Altman attack drama and the warehouse fire attack that happened recently, and I've been finding the reactions pretty scary. General sentiment on HN is something along the lines of "Altman deserved it" and even among my general leftish acquaintance bubble the vibe is along the lines of "they shouldn't have missed" or "we need more of this fuck the rich" which doesn't really bode well for the stability of society.
Whether or not you believe the more bombastic claims of AI CEO's, I do think it's clear that at minimum AI is going to exacerbate the trend of technology centralizing power, wealth and status, even as absolute material standards have continued to improve beyond the wildest dreams of 99.9% of humanity in the past. For better or for worse, human happiness seems to be tied only lightly to absolute material standards and heavily tied to relative status, position, and feelings of fairness, and the internet and social media are super-stimuli for the human sense of status calibrated towards the Dunbar number.
Ruling out FOOM levels of societal disruption, I can think of a few ways that this plays out.
Left-wing communist populist marxist social democratic total victory: public outcry reaches all-time highs, perhaps with some peasant revolts sprinkled in, and the AOC/Mamdani coalition gets voted in to dismantle the AI labs, big tech and the icky billionaires. Leaving aside the fact that this would annihlate the economy and living standards by proxy, I'm not really convinced that with mass internet and social media there's any gini index or amount of redistribution that would leave the status anxious public satisfied. First they came for the billionaires and then they came for the homeowners.... Certainly comparable democratic countries with half of the gini index of America are still constantly flooded with rhetoric about eating the rich.
Right-wing AI strongman technofeudal democratic backsliding: political violence becomes normalised as a part of day to day life and as a response, perhaps after a significant assassination or riot, a strongman or group of technocrats use the violence as an excuse to seize absolute power, abetted by AI in part or in full. The lumpenproles are kept under control via mass surveillance, drones and guns or killed off entirely. The worst ending, but one that seems depressingly realistic looking at the history of inequality and failed revolutions.
Nothing ever happens: whether mass unemployment happens or not, most people end up with sinecures or welfare to keep them relatively pacified. Social media and concentrating wealth inequality continues to make people miserable even as absolute material conditions begin to reach sci-fi levels, and competition for zero-sum goods like housing in desirable areas and prestigious educations and sinecures becomes even more red in tooth and claw in the vein of the East Asian countries. Political violence gets somewhat more normalised, perhaps to Latin American or 20th century standards, but it's limited to isolated incidents.
Generally I consider myself libertarian and think that billionaires are good, actually, but I do think that inequality and society's response to inequality is likely to be one of the defining questions of the 21st century. While Sam Altman is the most visible face of AI to normies, pure game theory dictates that technological progress will continue with or without the consent of any individual person, company or nation-state, if the capability exists someone (or something...) is going to be the one that holds those reins to wealth, status and power, and as long those reins are held then the holder will inevitably be the target of the green-eyed masses. I don't think we yet have the social technology to deal with this and it's not clear that we ever will; I've seriously been thinking lately whether this might be one way that the Fermi Paradox manifests.
Shirayuki2
6mo ago
I'd say I'm both simultaneously. I think it's unlikely that scaling LLM's gets to AGI, so I'm a skeptic in that sense, but it is significantly more progress in AI than I ever expected to see in the 2020's.
With that in mind it does seem likely to me that AGI is achieved in my lifetime, and I think if it does happen then humanity is doomed for all the old Bostrom/Yudkowsky reasons.... don't see what I could do about it though, so realistically it doesn't really change my life very much.
Shirayuki2
6mo ago
And certain professions like SEO slop writer, translator, and others are definitely disrupted forever regardless.
At least in the case of translators, I think you'd be surprised. I happen to be acquainted with a good number of professional translators and almost to a man all of them are still booked out in terms of work and make solid middle class incomes.
My understanding is that the "ChatGPT" moment for translation was around a decade ago when neural machine translation was first getting good. Already at this point, for translation tasks that didn't require professional-grade reliability or well-written prose, Google Translate or DeepL were basically already good enough; translation for things like manuals or brochures was commoditized well before transformers.
Of course LLM's write much better than DeepL, but in practice the set of translation tasks that can't be delegated to Google Translate or DeepL, but can be handled autonomously by a LLM, is actually quite small.
High-reliability translation tasks like legal, medical or diplomatic still require a human in the loop, and LLM's are still subpar at translation tasks that require a high level of interpretation, as in the case of literary translation. At a high level, a good literary translation can be thought of as a re-writing of the original work, and as of yet LLM's are still quite poor writers without significant human intervention.
Money is fungible. To a first approximation, if a trillion dollars is going into trying to build data centers (and failing, in many respects) that's a enormous chunk of resources that isn't going into fusion research, hiring, fixing bridges, what have you - things that could directly and meaningfully improve people's lives.
FWIW I'm optimistic about AI as a normal productivity-improving technology for improving people's lives, but this is explicitly not what the frontier American labs are aiming for, and it seems extremely difficult to justify the trillion dollar investments in a AI-as-normal-technology world where tokens are inevitably commoditized and primarily inferenced locally.
I guess I'm just feeling a disconnect from your posts. If you want to be a hard-nosed historical realist, OK: in the same way that the Treaty of Versailles and the Great Depression made WW2 inevitable and Hitler was merely in the right place at the right time, I agree that race dynamics ensure in no realistic world does a bilateral pause happen, and that as soon as Attention Is All You Need was published realistically no force on the planet could have stopped the AI race from accelerating - Altman and Amodei were merely in the right places at the right times to ride the wave.
But then you go on about "how can you not be excited" and "[bringing] about a new age of American prosperity" and I'm just confused about how you can think we're being forced into a world where if anyone builds it everyone (probably) dies and think this is a good thing. At best, if by some miracle alignment is inherent, you're under the heel of God-Emperor AI Frontier Lab Leader or God-Emperor Democratic/Republican President (adjust for your outgroup party) for all time and at worst all America's done is won the "created the Torment Nexus first" award.
At the end of the day "American" and "Chinese" are merely constructs of society as we know it and not immutable laws of reality - I fail to see how "superhuman general intelligence" and "liberal democratic All-American values as we know them" or "superhuman general intelligence" and "socialist values with Chinese characteristics as we know them" could meaningfully co-exist, so what benefit does the median person really get from ultimate, unaccountable power being instantiated 2000 vs 6000 miles away from them?
More options
Context Copy link