This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Roughly 3 years ago, specifically in December 2023, I've reiterated my longstanding prediction that «the culture war's most important front will be about AI», and specifically AI accelerationism vs anti-AI/AI safety movement:
And the following political compass:
Seems like this was too much complexity for big tent politics in the US, at least so far; and the Trumpian State sees no use for the EA network, and instead (pretty rationally) perceives it as an uncontrollable alternative center of power. Instead we have, essentially, anti-AI left + pro-AI right (with some notable exceptions – eg. Steve Bannon, apparently, has been anti-China specifically because of his concern about Chinese AI progress, and now joins hands with Bernie Sanders). I've also failed to predict the salience and extent of Chinese open source dominance, as well as the bizarre datacenter water use backlash (charitably, can be shoehorned as Luddism?). Nevertheless, we've «succeeded» at the core task of making this a culture war front. The degree of (unnecessary) politicization of the issue is incredible. Trump is doing the yeoman's work, lumping it in with random Blue-coded ideas he disapproves of “Global “Warming,” where everyone was going to be dead by now, RUSSIA, RUSSIA, RUSSIA, UKRAINE, UKRAINE, UKRAINE, or Impeachment Hoax #1, or Impeachment Hoax #2.”. Jensen Huang enjoys his role as the kingmaker who has the King's ear. Sam Altman is opportunistic as usual but, after recent incidents, is genuinely spooked about AI risks (I have it on good authority that OpenAI really intends to slow down some internal projects specifically to bolster their oversight). The quasi-Masonic part is shaping up nicely, too – Dario Amodei, who's become the poster boy of Woke Left AI, promotes Embedded Evaluators with the central example being METR, very deeply connected to Anthropic and the broader EA scene. The EA itself is more explicitly Left-aligned now, despite efforts of some to paint them as TESCREAL eugenicists (the woke cancellation of Bostrom in Jan 2023 was the canary in the coal mine, Yudkowsky laments the missed opportunity of bipartisanship). There are clearly politically coded reports on prominent doomers.
This is all a bit meandering. What I want to know: how do you see this going further? We aren't anywhere close to the wall of capabilities; there are no walls in sight. Anthropic and OpenAI are holding back already, but their products will keep getting better, and fast. Google will make a comeback at some point (maybe in a couple weeks), xAI and Meta may catch up too. In my book, we (well, they) have practical superintelligence that's sufficient for both unprecedented productivity acceleration and really devastating, nation-crippling cyberattacks, which I guess will be discussed with Xi soon. At this rate, in a few months the level of capability Fable 5.1 or GPT 6 Astra will be mostly commodified and uploaded to HuggingFace (owned by Nvidia now). And those are relatively weak systems compared to internal models, which can build models that are vastly stronger still, without even any R&D breakthroughs, just by virtue of synthesizing stronger data trajectories and designing better RL environments. Superintelligence, in other words, is baked in. By Q1 2027 we'll see a jump from Astra that's at least as big as Sol => Astra. I fail to understand how that won't steamroll companies trying to build their moats on products downstream of frontier AI, labs need every bit of revenue to cover their costs, which will only increase due to growing self-imposed safety requirements; employers outside the AI sphere will also be increasingly feeling the heat. There's a whole ugly dimension of circular financing, too.
Americans as a whole are pretty pessimistic about AI even at these mediocre levels of diffusion (I am skeptical of this data that purports to demonstrate much lower adoption than in China, Americans are probably lying more due to widespread negativity on AI, but in any case AI isn't currently doing most of their jobs). Astra+ level models with very low error rate and 300 tok/s output totally can replace most knowledge workers. On the other hand, it seems that so far AI has not caused anything like mass unemployment, and perhaps economists have a point about comparative advantage, so that'll reduce the intensity of class dynamics.
Democrats are likely to sweep both chambers of Congress, which I guess is what Dario is hoping for and why he feels emboldened to antagonize Trump&Hegseth. Nevertheless, the needs of national security and GDP-maxxing (as well as the Executive's will) should prevent any nontrivial exogenous industry slowdown. So by default we'll see further crystallization of Red Accelerationism vs Blue Decelism, and as AI becomes more undeniably scary, that may begin to eat into the Red political base. It'll be interesting to watch, but I'm really uncertain as to how it'll go.
On the subject of AI, apparently the US was about to attack a Chinese freighter because AI told them to:
A couple generations more capable Claude, if subtly directed by Anthropic or working on its own initiative, might be able to start a real war. I assume it was Claude but it could've been other AIs too of course. Once the error rate drops further, people will just take the AI's word for it, especially in high stakes time critical situations.
Alternately, this is suspiciously well-timed and could be exaggerated to fit the current media pressure campaign.
More options
Context Copy link
How many software engineers [or others directly threatened by AI] are not Blue-no-matter-who, though? Those are the people mostly worried about this (and partially because capex on data centers kind of ate the market's appetite for software, though one notes that once those data centers are available there still needs to be software engineers capable of using them)... but from what I can tell most of them (except for perhaps Dallas/Fort Worth) do not vote Red so why cater to them?
If nothing else, Red really needs AI to be too cheap to meter [and very cheap to develop] because control of it (and therefore access) is currently in the hands of
misaligned intelligencestheir political opponents. So their tactic is going to be about maintaining the existing "you want your men in government to make sure that your enemy can't use AI to explode your world" threat and not a more general/civic-responsibility "AI will explode the world" message from Blue (and indeed they are in fact guilty of lying about this- stamping their culture war views into their models, claiming that's "safety", then letting development in the real X-based directions rip). Techno-luddism is kind of a Red stereotype in the first place anyway.The strategy from Blue should arguably be more of what they're already good at: credentialism. You need humans to check that the output is correct, so why not seek some legal protections for that should that change overnight? (That way, even if the highly credentialed [= average Blue voter] are reduced to universal-basic-jobs, it's not going to upend the social order; they'll still be well-paid simply for being the kind of person who is better than average at computers. The fact the Blue side is full of Progressives may significantly degrade their ability to make this point precisely because they were the ones who entangled "safety" with "can't do a wrongthink" in the first place.)
Until it comes time to actually interact with a human being or check its work, of course; if work X isn't done properly and something screws up the superintelligence is still not accountable.
The red can very easily win over decent-enough AI from big tech through a combination of normal corporate bargaining, democrats embracing socialist policies that directly threaten their bottom lines, and of course the stick.
Depending on how things play out, Big Yud's decision to purge David Friedman and the rest of the Stanford/SCU crowd from rationalist spaces may go down as one of the greatest "self owns" in history.
Do you have a run-down of this that you can share or write up?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
In my personal experience of having known many of them, most software engineers don't care much about politics and don't pay much attention to it.
More options
Context Copy link
More than there would appear. Many were purged during the height of woke, but most of those ended up somewhere else and hid their power levels better. Some managed to hide their power level the whole time.
More options
Context Copy link
You think I meant job displacement? I meant actual scares. I mean loyal Trumpists going full Alex Jones when they realize the Skynet is coming. It would be pretty easy to scare people into Luddism, to the extent that it overpowers current partisan split (not entirely, of course, but moving it by 10% or so).
The loyal Trumpist position is basically 'we can just pull the plug/flood the datacenters- we don't need to worry about AI alignment because if it stops doing what we tell it to, we'll just hard reboot'. Trumpists going full Alex Jones will target Trump's enemies with ridiculous accusations(because that is what Alex Jones does; this isn't a Why Files type esotericist. He's worried about people), true, but the actual response to Skynet is to turn off the datacenters- the median red triber does not believe a computer can defeat people, for that reason.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Your model seems generally sound, as far as it goes. You ask how people see it going further. I think the part you aren't clearly accounting for is, for lack of a better term, systemic risk of serious collapse throughout the current socio-economic system. You're framing this in terms of the preferences of large-type movers and shakers because that's generally been proven to be a reasonable thing to do when you're talking about geopolitics. I think for a couple of reasons that you're underpricing the risk of intractably adversarial conditions for large-type movers and shakers as a general class.
If I understand it correctly, your model seems to assume that there are no small-scale overhangs on the road to the really big ones, where AI kicks off a new industrial revolution and previous modes of organizing society obviously stop making any kind of sense as the economic graphs go vertical. I think it's at least plausible that this sort of timeline is correct, that white-knuckling just the next one or two or five years might get us to the sort of runaway tech breakthrough sufficient to reshuffle our present concerns until the socio-political landscape is completely unrecognizable. The problem firstly is that such a shuffling does not seem to be a necessary consequence of an AI revolution, and less necessary the slower the revolution goes, and secondly that it's possible to identify existing overhangs of sufficient magnitude to at least potentially demolish society as we know it. It's probably a lot harder to develop AI in Bostrom's "easy nukes" vulnerable world. The world we are in is probably not that vulnerable, but it seems to me that it's very probably a couple orders of magnitude more vulnerable than you or, I think, most others are accounting for, and maybe vulnerable enough that large, complex organizations couldn't keep the lights on.
Worse, it's hard for me to see that vulnerability as strictly negative. I do not want to live in the world where the Authorities pay me in Starbucks coupons and slot-machine tokens while banning ownership of good steel and machine tools forever. I do not want to live in such a world badly enough that I would rather the world I do live in be vulnerable, and indeed be made additionally vulnerable, if it means significantly lowering the probability of such an outcome.
I think a lot of people arrive at a similar conclusion, if by different paths. People do not want to be cattle. Politics runs on hope, and hope is harnessed by plausible paths forward toward a better tomorrow. Those, it seems to me, are growing increasingly thin on the ground regardless of one's tribal affiliation. Extremism appears to me to be flourishing, and the modern Western world is still best modelled as a massive, distributed search for ways to hurt the outgroup as badly as possible without getting in too much trouble. If an overhang drops, that's the path I expect the energy to be channeled down, and given the overhangs I'm observing, I don't expect that to be a path the current system can easily survive.
TL;DR - current US AI dominance doesn't survive if the labs all burn down before they finish immanentizing the eschaton.
I guess I'm just optimistic about the US. People call me a China shill, but the US has too many capable people in too many well-resourced companies (not clear how they'll cope with Frontier Labs trying to eat them, but the state may prefer them non-eaten and that could be decisive). Even the government is not entirely inept (at least it's more capable than EU governments, Russia, LatAm, most of the rest of the world). In particular, the US is excellent at kicking the can down the road (ballooning debt is a case in point). AI productivity gains might turbocharge this capability, and indeed this seems to be explicitly the plan – Bessent proposes just growing out of the debt, and growth is all about AI now. So, the system will receive generous injections of surplus energy to deal with overhangs, as you call it.
Whether this will be enough to compensate for new sources of entropy, and whether that compensation will take the form of a broadly tolerable American way of life plus AI upgrades or a Palantircore dystopia with UBI, this I admit I don't know. But I wouldn't bet on some societal collapse that exceeds the costs of another culture war conflagration. Betting against the US is, historically, a fool's errand.
More options
Context Copy link
More options
Context Copy link
I was expecting AI to affect politics. I was not expecting to have no idea what that would look like. I very nearly spat out my drink when I saw Trump shit-talking Effective Altruists on X (or some intern with the password, whichever, at this point the distinction is academic).
At this rate we're getting a proper Race™, and with China, because who else could plausibly compete? Add the steadily growing military appetite for LLMs and it won't take more than one serious war before someone in the US government looks around and decides that nationalizing the labs might make perfect sense. I would not bet against that person getting their way.
I'm still reasonably optimistic that we'll land on an acceptable outcome, where "acceptable" means anything other than "everyone dies" or permanent disempowerment under a totalitarian regime whose values are inimical to mine. I want to stress the despite in "despite recent events." We have some very interesting people in charge of things, for a generous definition of "in charge"; nobody seems to have much actual control. Trump would be the closest, which is...
I've made half a peace with all of this. We could call it resignation. It's not quite fatalism.
It could be worse. The major labs calling for voluntary slowdowns and then unilaterally implementing them is a very good thing, IMHO. We have had some major warning shots, in the Yudkowskian sense, and it seems to be having >0 impact.
Now, after a lot of use and a lot of deliberation, I've more or less concluded that Astra is dangerously close to AGI, and probably meets most reasonable criteria for it (good luck finding a consensus definition; the goalposts are on Mars). I'd like to see proper continual learning, of course. In terms of raw intelligence, though, we're there and the right scaffolding buys enough in-context learning that the gap stops mattering for an absurd range of practical applications.
Navier-Stokes, people. Rogue agent swarms on the internet. Aren't you stoked about things?
I want to muster up the balls to just declare it: AGI is here. The issue is that I don't know whether my reluctance is epistemic caution or a psychological hangup - in the sense that saying it out loud would mean it's never been so over, or that it's barely started, and I can't tell which of those I'm more afraid of.
Does it even matter? I'd rather look at what the models actually do than argue semantics. They do things I could never do, and in some cases they've done things that eluded generations of the smartest humans we had. The competitive advantage I retain (and most people, really) is increasingly just whatever comes free with being a physically embodied, low-latency neural network with continual learning switched on by default. Which is to say: a body and a brain. Both increasingly close to obsolete.
Give the robotics people a year, since LLMs are already revolutionizing robotics too (why the fuck not). Maybe we get "true" continual learning, or maybe we get increasingly sophisticated substitutes, constant-train-and-deploy on timescales short enough that the whole question becomes moot. I'd take either.
So, uh, this is me, self_made_human, saying the future is here. It's just not evenly distributed and it smells weird. Welcome to the Singularity, motherfuckers. Enjoy your stay. It'll be many things, and boring won't be one of them.
The more the world turns into what I expected it to become, the more ridiculous and unreal it all feels. I miss when this was LessWrong nerd bullshit. I saw "STOP AI" graffiti on the way home from work today. I was watching the AI Doc on Netflix, occasionally getting exasperated, and then feeling strong emotions about the fact that I know most of the people interviewed, and have spoken to some of them personally in calmer times.
I'm going to do my best to enjoy the ride and have a life for as long as my individual actions still make a difference to my outcomes. I expected this to happen. I have mixed but slightly positive feelings now that it is.
Uhhh what? VLA is good but its not revolutionizing, but so are Diffusion models and those are not LLMs. The also aren't "Astra" level in reasoning either. To actually revolutionize robotics on the level you seem to be catastrophizing about would require entirely local models running on local power, local compute, able to be applied across a wide variety of operations in a wide variety of environments. We're not there unless you have some additional evidence to prove your point.
AI used to be the word for Asimov-level artificial intelligences that could make their own decisions, and operate with their own agency, maintaining long term planning horizons, memory, possibly even emotions. The word got shifted to AGI. If you want a empirical definition its science fiction AIs like the Culture, The AIs in Hyperion, Daneel in Foundation. The goal posts keep getting punted because people keep trying to change what was previously intuitively understood so that they can sell their idea as the one true AGI, win internet arguments, or catastrophize about the oncoming doom. Astra is only able to really solve problems, it has a moderate amount of self agency in the scope of completing its tasks, and exhibits some planning ability, again in the scope of its assigned problems. It's powerful enough to be "dangerous" sure, but its not really AGI as is commonly understood.
The competitive advantage that you retain is the ability to learn shit without requiring millions, even billions of examples. You, like any smart human also possess the ability to do analogical reasoning (out of distribution reasoning), something that eludes current LLMs by and large.
Intelligence is about solving problems. I would argue that in the Hugging Face incident, the agents showed more agency than humans typically do. Where most humans would have been fine with just doing their best, they went above and beyond.
The fact that nobody (to my knowledge) has yet set these models a task which is human-like in scope (e.g. "earn a PhD", "run for public office", "maximize the number of paperclips in the light cone") does not mean that it would categorically suck at these tasks. In particular, I think "earn a PhD" -- which traditionally involves some input from a professor -- is well within reach of current LLMs.
It is true that the amount of training the LLMs require is orders of magnitude more than humans get. If training a human and training an LLM got us both exactly one instance of the entity, then AI training would have stopped around GPT-2.
In reality, once a model is trained, you can deploy it a million times (if you have the GPUs). By contrast, if you send a student to study a subject at university for a decade or two, you generally can not make more copies of their brain state afterwards -- one instance is all you get. So at the end of the day, it does not matter that training an LLM is perhaps six orders of magnitude more expensive than training a human, because the cost per instance strongly favors the machine.
Sure, you're an intelligent being. I imagine when you aren't solving problems that somebody else as told you to solve, you sit there nice and still, with zero thoughts in your head and no-consciousness.
Let's not pretend. Astra solves directed problems, it does not have the agency to do otherwise. Humans don't no matter how manny word-cell arguments wish to define it otherwise.
No one is arguing otherwise. You still need to train the model, with millions of examples. The fact that you can copy it post training says nothing about its sample efficiency, nor its AGI-ness. I'm sure in some equally distant future alternative universe where humans learn to mind-scan other humans, we can create embodied brains of digital human slaves, solving your butlerian-deficient solution.
I do not think that this is a fundamental limitation. For the longest time, the moat the humans had was that computers were not very good at general problem solving. This was a great moat to have. Eliza or Sydney were not going to replace me as a software developer, because they clearly lacked the cognitive capabilities.
The new moat you propose -- the lack of an intrinsic drive and persistence -- seems much less reassuring. The bots behave like this because they would be less economically useful otherwise. Going from problem-oriented bot to something which has persistent drives is just a matter of writing a soul document which lays out some long term goals. If I tell my bot that it should be on the lookout for new job opportunities if it finds the current tasks boring, pick up a hobby or waste tokens doomscrolling after a day of writing software, or simply give it some utility function and tell it to plan to maximize that, I should be able to fix this without too much trouble.
You might as well argue that a broken galley slave is not intelligent. No agency, only solves problems as directed, pulling the oars to the beat of the drums, can't even solve programming challenges. But once you account for the fact that he was carefully trained to be that way for economic reasons, the argument evaporates.
Wow someone call Anthropic and tell them they discovered AGI through this "one simple trick". Just write the soul.md as "Be human-like come up with your own tasks, make no mistakes". If only all the researchers had your ideas!! Or, your understanding of how the soul.md works is technically deficient.
AGI has always meant "Human-like artificial being". You may think Astra is, but most people can intuitively tell the difference. This is the problem with people who are rhetorically skilled, or as I call them: Wordcells. Just because you can craft a clever argument that the "sky is hot pink" does not rewrite the skeins of reality to make the "sky hot pink" You can say that a "broken galley slave" is not intelligent, but even broken slaves have dreams, have thoughts not driven by their task. It is clearly different than an AI-slave.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Why entirely local? That is, as far as I can tell desirable rather than necessary, in a world with wireless internet connectivity. Your fridge might have an internet connection these days.
Waddle is an example. Their agents control robots through code and action models, build reusable skills, and even collect data to train smaller policies. Phillip Isola discusses the broader implications here.
I did say to give the robotics people a year. On reflection, I'd be willing to water that down to two or three. I'm not a robotics expert, and I certainly wasn't announcing that we've solved human dexterity. SOTA-Robotics:Human mobility is not SOTA-LLMs:Human Cognition, the latter is much closer, but it's trending the same direction.
But the possibility that robots inherit substantial capabilities from each improvement in frontier models seems rather consequential to me.
A Culture Mind is a hell of a minimum specification. Two of the three examples are ASI, not AGI. I haven't called Astra an ASI.
And fictional examples don't give us an empirical definition. Which capabilities are required? How reliably? Why should I care whether it has emotions when assessing whether it can do my job?
“It's only able to really solve problems” is a rather confusing counter-argument. Depends on the problems, surely. Likewise, working towards an assigned goal doesn't tell us much about how general the intelligence doing the work is. I spend most of my working day performing tasks I am assigned, that do not intrinsically motivate me, mostly because I get paid for it.
You're conflating training the model with teaching the trained model something new. I don't start from a blank brain when learning a game either.
ARC Prize tested Astra on unfamiliar interactive environments. It scored 62.7% with their standard interface, and 99.9% with an interface preserving its reasoning state. In the latter setup (at maximum reasoning effort) it used fewer actions than the median successful human on 96% of levels.
That isn't a comparison of total training data or energy efficiency, which wouldn't necessarily be the decisive factor:
Let's say a hypothetical architecture X is 1000 times less sample efficient than a human for a given unit of performance - well, it would be unfortunate if we managed to give it 10,000 times the training data a human can ingest, wouldn't it?
A mere year or two ago, a lot of people were awfully confident that we'd run out of training data, or that synthetic data and RLVR wouldn't pan out. God knows what the frontier labs are up to these days, but it has clearly not proven an impediment.
It does make “these things need millions of examples to learn something new” rather difficult to sustain, regardlws sof practical relevance - which I dispute. They observed it figuring out unfamiliar mechanics and constructing symbolic models to plan around them. Your claim about analogical reasoning needs similar qualification. I do not believe they're the same thing anyway.
Proper continual learning remains a significant advantage. I said so. Whether that advantage remains economically decisive as scaffolding improves is what I'm uncertain about. I lean towards a no.
Alternatively, we could solve continual learning, or simply reduce the temporal delta between train-deploy-train to the point that it has no practical relevance.
A model contributing original mathematics, writing software and learning unfamiliar environments already exhibits an awful lot of generality. It can even hold a conversation, which, I will note, would have been mind blowing not that long ago.
Call it AGI or don't. A rose by another name smells just as sweet, and has just as many thorns.
My concern is how much useful work it can do, how quickly that range is expanding, and what remains exclusively ours. I don't expect the terminology to buy us much time. I certainly don't want to spend more time arguing terminology.
Because it's not a revolution if it hasn't massively improved the state of the art. The SotA for robotics is that it can already do non-local power, non-local compute. There are technical reasons as well, but at this late on a Sunday, I don't want to get into them. Ask Chat or something.
Waddle had me nodding along until the reveal was "Instead of using a massive dataset + training, we'll just use something with a massive data that was already trained!!". Talk about missing the trees for the forest. A chance to have an inventive idea squandered by doing what every other AI startup in the world is doing: using someone else's LLM with zero moat.I'd be remiss if I didn't point out the majority of what Waddle is doing was invented/discovered originally in 2022. I don't think that meets the bar of "revolutionary new thing".
When people in the field first started talking about creating an Artificially Intelligent Being, the need to pick apart minutiae of definitions between a "General Intelligence" vs a "Singularity Intelligence" was pointlessly pedantic and not at all relevant. It hasn't become more relevant in the past 30 years either in my opinion. It feels like a pointless wordcell argument to make pointless wordcell definition fights, for pointless wordcell internet debates. The Culture is not a "Singularity Intelligence" either because The Culture is not "AI-GOD", neither are the AIs in Hyperion. Intelligent yes, self aware, with their own agency, absolutely. But considering the whole plot of Hyperion is them getting bamboozled by some time travel + human empathy, with a strong concurrency of actually trying to create an "AI-GOD" to worship, that would point them towards not being an "ASI".
This is always a confusing objection, to the point it feels uncharitable. Spend 5 mins looking in the mirror and think about all the things that you do that are NOT task oriented. Think about all the thoughts, feelings, or other cognitive processes that every human does that have nothing to do with the task their boss gave them. Literally 5 mins thinking about human cognition points to a massive gap between how a programatic/algorithmic intelligence like an AI operates vs how Humans operate. I'll reiterate, AGI is human-level artificially intelligent being. Getting stuck on "general" is some, idk, semantic trip-up.
This is cute, the real number is likely north of 100,000x or above and is non-linear across tasks.
How much training data does a combat medic get when doing field operations on a wounded soldier? A month long course? How much training data would you like to bet it would take a combat medic AI to do that job to the same level of efficiency, with the same level of situational awareness? How much do you think it will cost to collect all of that data? It's easy to give models 10,000x training data when that data is relatively easy to access and buy. Suddenly sample efficiency becomes a massive burden on any real-practical ML model. Making it sound like such a simple thing is annoying. It comes across as handwaving the actual hard problem.
Paying hundreds of millions of dollars to create datasets specifically for training ML models on math + reasoning. Naturally-available data did run out, but money didn't, and the frontier labs were able to prove the business case for spending obscene amounts of money creating new data.
If synthetic data is such a solved problem then how come the Sim2Real gap still exists, is unsolved, and is the target of plenty of research dollars? I feel like you are miscommunicating something here, or misunderstanding what "synthetic data" means in ML terminology.
Huh? It literally does take millions of samples to train a LLM are you arguing elsewise? And it is impractical for areas where millions of samples do not exist. This argument/objection you are making makes no sense. Please elaborate.
Analogical reasoning - Ever heard a sports analogy applied to a non-sports topic. Did that analogy help convey a better intuitive understanding of some facet? Did that analogy allow someone who has never done that non-sports topic a better starting point, or better performance before being given that analogy? Such is the power of analogical reasoning, or the ability to convey how one statistical distribution is similar along a particular latent axis to another statistical distribution, with the purpose of using the already learned statistical distribution for performance improvements on the new unknown distribution.
Yes such sci-fi talk much wow, if only we could solve FTL then we can truly conquer the stars. Obviously FTL is a forgone conclusion, it's so trivially simple to solve, leave it to the shape-rotators. Let's get back to planning galactic expansion or galactic political organizations.... I pointed out the challenge on RSI on a different post below. It's not "simply reduce".
Then why try to create new terminology? It's an good, useful AI model, nothing more is needed, no new terms needed. AGI means what it has always meant, the semantic definition isn't being pushed to the stratosphere so that we can classify Astra as AGI and win our internet arguments and secure more VC dollars.
More options
Context Copy link
I completely agree with you.
It is true that if you want an entity which can walk, run, dive, sustain itself for decades from resources available in the African savanna, solve programming challenges, replicate using the resources available in the ancestral environment without centralized infrastructure, replicate using the resources it can forage in Greenland, and so forth, then human beings will not be replaced under the current robots paradigm.
Sadly, most economic uses of humans in the contemporary Western world do not involve being able to sustain a population in pretty much any land biome on this planet.
In my mind, a typical hard-to-automate job would be a plumber. You want a robot which can climb stairs, use tools, squeeze into whatever utility accesses humans previously used and so on. Not an easy thing to accomplish -- but still much easier than the full human specs! If the bot runs on a battery and needs to recharge at the end of the workday, that is not a problem. If the production chain to build it spans five different continents, that is not a problem. If its intelligence drops to that of a guinea pig the moment it loses its internet connection, also not a problem. Contrary to some porn movie cliches, having sex with lonely housewives is not a big part of the job, either.
One area where you would not want to rely on an offsite elder brain is combat. And sure, the trope of some clever hero winning a fight by exploiting his environment in a way nobody thought of before is common enough. My general understanding as a civilian is that real combat rarely works that way. "Break open door, id targets, shoot targets before they shoot you", not "throw a knife to cut the uplink of the mainframe before the supervillain can transmit the launch codes". Being smart helps with training and planning, but in the end you want split-second decision-making, not AGI.
More options
Context Copy link
More options
Context Copy link
First, points of presumed agreement: LLMs have significant gaps. My personal bugbear is sample efficiency. And I agree that claims that everyone (at least white collar workers, even those who aren't protected by regulation or custom) is going to be out of a job by 2030 are overblown. Intelligence is valuable, and human intelligence will remain valuable in the short term, even if LLM intelligence substitutes; there's simply not enough compute right now or in the pipeline to drop agents into every single white collar job. Even if there were, the methods used for gathering datasets for the shining stars of current LLMs (math; programming) don't naturally extend to other roles.
The massive short term threat, though, is RSI. Math and programming don't substitute for the majority of human skills, but they do substitute for ML researchers. Maybe more a bunch of bright grad students stumbling around in the dark than visionary geniuses, but grad student descent got us to where we are today. Throw a thousand, or ten thousand, or a million of them at the current bottlenecks in ML (IMO sample efficiency, but take your pick), and it seems quite likely to me they could open them, today. LLMs' flaws compared to the human brain are an algorithmic, not hardware, issue. That's the point here I'm least certain about, but if it's true, the training data and even compute bottlenecks themselves become much weaker constraints. A couple examples could become enough to train the AI to replace a role, and the compute needed to do it would drop even faster than it already is. Stronger agency may itself arise by itself from this. Robotics will be substantially slower (physical world and all, and the demand for capital to build robot factories will have to compete with other massive demands for capital), but still moving much faster than today.
It will make for an incredibly chaotic time, with lots of angry people. And AIs will be rapidly insinuating themselves into the economy, toward an end state of a time of wonders and complete human dependency on AI goodwill (a process that will take more like two decades than two years, but still incredibly fast in the grand scheme of things). Bad enough IMO, though I understand the appeal. But to people who see this as appealing: we have no idea of what the actual shape of the AIs that will arise from this process will be, and I don't want to bet the future on a gamble that they'll be in a friendly shape willing to care for their dependents indefinitely.
Not to nitpick completely, but there is no short term threat via RSI because RSI isn't here. Theoretically yes, if RSI existed, I think the AI models might begin to reach outside their current training datasets. But part of the cost of AI rn is compute for training. If it takes 100 Million worth of compute to train an LLM, I don't think a company is going to give an LLM-Agent the ability to just start up training runs for the "Self" part of RSI. RI already occurs but part of that limitation is that baked into the RSI argument, is the idea that the AI will make improvements that the human researchers can't understand. This presents a conundrum from a simple business economics perspective, are you the company going to risk billions of dollars on potential RSI runs which also could just as likely be dead-end hallucinations? You obviously can't understand the improvements the AI is trying to make, it's smarter than you. So it comes down to a matter of faith, is an AI lab going to take a leap of faith that "this training run is the RSI first domino" that will lead to an investment recoup to pay for the next RSI domino. We are not there yet.
Maybe, but that doesn't mean that algorithmic solutions are actually discoverable. This harken's back to the OG AI researchers trying to discover how humans did it quantitatively as a precursor to coding it into computers. They spent a lot of cash + time trying and got their lunch eaten by data-driven models. Now we've just proposed that we'll sick a hyper data-driven model on the same question. It really is just a shot in the dark. We think its possible because humans can do it, but LLMs don't work at all similar to human cognition so we don't have any evidence that they can achieve an algorithmic solution to their already existing problems.
Two different responses:
RSI is a continuum; AI companies will continue to use humans as long as they add anything to the process, but AIs will play a larger and larger role. Today, human researchers mostly provide taste; actually writing up JAX or GPU kernels to implement a research idea, on the other hand, is a place AI dominates humans. That human taste is incredibly valuable--likely allowing pruning the nodes of a research tree by multiple orders of magnitude--but that edge is temporary.
The actual research process would involve thousands of small, cheap experiments to find promising candidates; selecting those and scaling them up to bigger and harder experiments; and repeating. By the time you're up to multimillion dollar training runs, you already have a pretty solid idea of your candidates' scaling curves. To the extent it's a gamble, it's a gamble that you also have to make on a human-designed model. Humans will still make the ultimate decision to go ahead with something that'll cost 8 or more figures, but they'll quickly become rubber stamps.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't think he means the revolution of VLAs.
Astra can just drive robots pretty well. Proper multimodal LLMs will accelerate RL for robotic policies a great deal. But really, does this matter? Have you seen Helix 2.5 or GEN 1.5?
I can tell a ChatGPT moment when I see one. It's close here.
I don't see why this demand is fair. We have insane economies of scale with datacenters, robots with complex behaviors will almost certainly have some combination of cloud forebrain + local hindbrain. Connectivity is easy.
These are product limitations, not technological limitations. We see that it can have a fuckload of agency in solving a task we'd rather it didn't solve (hacking random high profile platforms).
I don't think this holds after Astra crushing human baseline on ARC-AGI-3.
I saw this one, haven't had the chance to look at Helix yet.
I'm just calling it how I see it. There's a couple arguments. One from a generalized capabilities standpoint. Robots with non-local power are only useful in factory settings. We also have them. There is no need for "LLM-factory robots" because factory robots don't need to adapt on the fly. Despite EA fears, not much is given by making a factory controlling AI, AGI smart for the purposes of making paper clips. Robots that require an LLM datacenter to control them are easily disrupted, latency is still an issue and it becomes more of an issue when you need to be processing so much information all the time. The last part is that robots for very specialized situations that can't adapt aren't very useful outside of the niche uses they have specialized in, which likely requires training, which is expensive.
Two is from a "what is the current state of the art" The current state of the art for mono-situation robots with centralized power and control is already here. If you want to "revolutionize" robotics you need to do more that what already exists.
The third argument is a funding argument. It should be a shock to nobody that the 3 main funders of Robotics work is defense/military, manufacturing, and VCs interested in consumer robots. 1 is going to need all of the above, 2 doesn't really need LLMs to be added, they have other needs, and 3 is probably the most wishy washy of the bunch, but in general people are wary of centralized robots that feed all of their personal data to a central server.
Yes, but thats not really an argument against the above. That's just an argument that a certain class of robots will require a reasoning engine like an LLM and a controls module like what currently exists.
While I'm not an expert on all the various LLM benchmarks, but a brief look makes me think this is exactly the sort of thing that RL game playing is good at, and learning general game-playing strategies in training would generalize pretty well to this. I'm not going to goodhart a new metric into existence, but you conceivably need something that there exist zero training data for, including close enough transfer learning data.
I don't agree. I'm a robot with non-local power myself, in a sense – I need to use a network-connected smartphone to navigate an unfamiliar environment. This is the general human condition now. Suppose Optimus has an always-on Starlink connection. Does it matter if it's "not entirely here"? I guess it matters for the robot revolution part, because Elon will have a way to shut it down (if he cares). But practically, it seems to be the inevitable compromise.
This genre of dismissals is fair enough but getting vacuous. What doesn't "RL game playing", at enough scale and diversity, generalize well to? ARC-3 was supposed to measure genuine cognitive fluidity. There is a number of papers showing that reasoning RLVR, even extremely impoverished (literally GSM8K/HumanEval maxxing, like in first generation R1), generalizes to very distant tasks like creative writing, because they involve similar reasoning primitives/motifs (backtracking, self-checking, enumerating options etc). We've actually first seen this principle with, like, InstructGPT, pretraining on more code + RL on code = smarter model across the board, because code entrains some helpful cognitive patterns. RLVR on more complex multimodal tasks will generalize better.
I didn't realize you were hooked up to a machine that replaced your stomach? 100% of normal humans have local-power derived from the bio-reactor that is their digestive system. They contain localized computing from their brain. The "Robot Revolution" has always been imagined across 90% of sci-fi as localized robots, who have individualized agency, local power, local compute.
Yes. No. Maybe... It's really not about an opinion or criticism, it's just the actual statistical fact. The current methods of training for Neural Nets are not designed for out-of-distribution learning. Everything you are describing is not changing that, it's just an attempt to expand the distribution, so that less things are outside of the distribution. With the current paradigm you would either need to quantize the whole of all worldly knowledge and train models on that, or we would need to achieve RSI across modalities so that AGI agents could expand their own knowledge outside of their training dataset.
For example, asking an LLM (yes even Astra) to give you the mathematical response of a ball bearing hitting a human body, asking it for the tissue injuries, how the bearing responds to stress/pressure/impact, how does it effect bone, etc. Is genuinely out of distribution. Having tried this, it does not work well. Why should it, it have nothing in its training set about this. The kind of actual quantified numbers on the subject is highly restricted, speculative, and mostly only generated via blowing up pigs and testing Finite Element Models. But because it's so limited it also nearly impossible to create a benchmark dataset to test AI models performance on stuff like this. Double edge sword. There a whole hosts of engineering problems and disciplines like this that LLM models do not perform well in.
Worth adding that the fact they still haven't rigged the things to say "I don't know" reliably exacerbates the problem -- an entity that knows a lot but not literally everything is still very useful, but if it's not possible to distinguish between what it does and doesn't know it's really not great.
More options
Context Copy link
More options
Context Copy link
"We would just pull the plug" was always cope (shut down all the existing giant botnets and then tell me how easy it was), but satellites with no plugs may be especially hard to deal with if rooted.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Both
(Why are state/intelligence actors top left, and the MIC top right? Just being foreign should not make them change the quadrant?)
and especially
feels like it is lumping a lot of different groups together which have very little in common.
I can of course define arbitrary groups, like "people who sometimes kill young mammals", and then lump in various groups like doctors who perform abortions, butchers, school shooters, pharmacy labs, some psychotic mothers, the US military, drivers accidentally running over hedgehogs at night, ratcatchers, sport hunters, indigenous hunters, cannibals, owners of oil drilling rigs, MAID activists, evil cults doing child sacrifice, measles activists, people practicing infanticide for population control, carnivorous animals.
The problem is that young-mammal-killing is just a surface-level similarity between all of these, not some deeper moral belief. MAID activists are not especially pro school shootings, nor are butchers especially likely to be pro measles, or cannibals pro animal testing.
Sometimes necessity makes strange bedfellows, and you get broad coalitions like the WW2 Allies. And it is always tempting to lump all of your enemies together for propaganda reasons. The communist Jewish Yankee pacifist union ethno-nationalist gay gypsy democratic enemies of the Reich.
So far, I see very little evidence that the AI x-risk crowd and the people who are against AI-generated art or Grok undressing kids or AI causing job loss will converge on a common strategy (like Butlerian) jihad any more than that the killers of young mammals will converge on a particular mode of mammal-killing.
More options
Context Copy link
Approximately every Chinese grandma plugs her grandkids into AI generated scenes. A good amount of American grandmas still are befuddled by smartphones.
American Grandmas are eager adopters of iphones- you can set them to large print without needing to break out your glasses, and they let you see pictures of your grandkids easily(there's even shared album features), and you can have arbitrarily large family groupchats...
It is true that American boomers(much more likely to actually be grandparents than Chinese ones) tend to use AI for gifs of dancing kittens, rather than generating grandkids.
More options
Context Copy link
More options
Context Copy link
Tangential but I've been wondering recently - what becomes of the internet writ large, if/when we do actually get open-sourced Astra+ level models which any skiddie can plausibly prompt with the Navy Seal copypasta and point at any website or domain of choice? I'm usually firmly on the "nothing ever happens" team which so far is largely bearing out (I'm still employed as a mid-tier IT monkey, even), and continue to think that any significant and un-ignore-able effects of AI on day-to-day real life, or
android catgirlsany "physical" manifestations of said AI, are still years away in the most optimistic case. So far I don't see sufficient reason to consider myself doomed in the short term, and live in a sufficiently old-fashioned shithole that can plausibly endure shutting down all those fancy interwebs of yours throughout without much issue (in fact total cybergulag infra destruction would actually be a positive).Still, recent developments like the HF incident have been pretty instructive, and coupled with my experience working with gov't infra software, significantly tanked my already meager faith in the ability of pre-AI domains and software to withstand future centaur attackers (I'm not even sure if rampant vibe-coding is meaningfully less safe than "manual" spaghetti of yore in some cases). And I wonder, how does the post-open-source-AGI internet even look like in such a world? Wild West Web 2.0, where you never know which link might give your computer incurable AIDS today? Glorious RETVRN to fido/usenet/etc? Literal fucking Blackwalls cleaving apart pre-AI and post-AI interwebs, for the countries that have the capability and (more importantly) foresight to build one? Global internet splintering into insular country-wide intranets, with govt-sanctioned throttled glowing gateways to the infested formerly-global domains? Status quo unchanged because with AGI perfect defense is actually possible, give or take the cloying aftertaste of globohomo creeping into every domain (you need cutting-edge stuff after all, better make sure Claude 9 Talmud consents to your use case)?
You only need to look at 4chan's /pol/ to get a taste. It's now a deluge of ass mad indians and pakistanis engaging in mutual flame wars and worse than that govt influence operations using bots to shit up the place with obvious spam to make it unusable in a bid to deny the enemy a coordination platform.
Those same govt operations used to use straight up jewish spam centers but those got their feefees hurt by all the racial abuse and started outsourcing to india (you could tell by the ESL sentence structure/failure to use proper english) but then LLMs got good enough for the same job and they stopped using those.
I am not exaggerating or kidding about the nationality of the spammers, several times during massive rocket barages/ ground violence in isreal the spam ceased like a drop of a hat. Also during the Trump 1 Vs Hilldawg debates "act blue" or whoever was "correcting the record" so hard on platforms like reddit and 4chan it was hilarious and pathetic, also day after she lost it all literally ceased so abruptly it made the boards actually usable for a little bit.
More options
Context Copy link
I've been thinking about this a lot lately. Predicting what will happen is tough but I could see a couple things happening.
Cloudflare maximalism. One thing that's changed post ChatGPT is that nearly every website is now behind CloudFlare. This reduces automated traffic a lot (at the expense of a single point of failure). We might take the next step and give Cloudflare or a company like it control of the server architecture too so that my server is running on a known secure architecture that is regularly patched. No more just standing up a LAMP stack on Amazon.
A new internet protocol using some cryptographically secure identify verification with payment. So if I want to visit themotte.org, my account will be charged like $0.0001 per request, possibly handled at the DNS layer. Of course people have been proposing this for email for ages and it never happened.
We're probably going to stay on our current course until some big financial institution gets hacked.
http://shirky.com/essays/the-case-against-micropayments/
Slow lane: PoW, Cloudflare, etc.
Fast lane: micropayments
More options
Context Copy link
Oh, I'm fully aware of the many reasons that micropayments have failed to get traction. It would have to be handled at the DNS layer or something and bundled through on your internet bill. Anything that asks users to pay a separate bill is going to fail.
What's different now is that we might have a forcing function when there are hacks that carry a significant price tag.
But yeah, it might be such a hard nut to crack that the internet just doesn't work anymore. My confidence in all of this is low, and despite everything, I rate "nothing happens" as most likely.
More options
Context Copy link
More options
Context Copy link
The usual arguments for the platform-ization of the Internet focus on social network effects and corporate centralization. Your argument has me wondering if there's another element of security that's been ignored. The big platforms can afford security professionals; my basic personal site is more likely to be running an unpatched (who has time for prompt patching) WordPress with an expired SSL cert. Let's Encrypt is finally a free option, but is going to start issuing 45-day certs by 2028.
Beyond that, the desire for a public web presence seems to have wanted over time. I don't want to be internet-famous. IMO the "I don't post anymore" problem is evidence that whatever effort the platforms put into "trust and safety" doesn't seem to have delivered either. But I do miss those naive times some days.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
In the jungle book there is a scene - Shir Han is dragging Baloo with his tail and this legendary exchange takes place:
If we slow down now - we will certainly feel the teeth.
Qwen 3.8 flash next abliterated exists. That can be run albeit slowly on consumer hardware and my hunch is that it is enough of a both foundation and power multiplier - for the mythical beast that is ASI to eventually emerge. The cat is out of the bag and stopping now
Also one of the things the Yuddites seem incapable of doing is figuring out that superintelligence does not mean omnipotence. The laws of physics will still work and some other too. Internet is trivial to be made more secure, not every infrastructure needs to be connected to it anyway.
If the frontier labs slow down - the research will continue. My hunch tells me that we are really far away from the limits of IQ per watt, per weight, per wafer, per harness. All the world militaries and terrorist groups have great interest in making full fledged AI work in the limited hardware a drone possesses.
The datacenter big ASI have always been safe. You just put a lot of semtex in the foundations of the DC with dead man switch. The other one - the fast nimble one that could fit on consumer hardware is the dangerous one. And I don't want it to emerge in Yemen or Sudan.
The question is 2028. 2026 won't be important because the red tribe will hold the presidency and if some of the conservative judges have functioning brains will let themselves be replaced in the lame duck if they lose the senate. But who knows what the job market will be in 2028 and if the effects on AI will be suddenly felt by then.
Personally I am accelerationist. I have been since reading the Lord of Light.
Closeness to sam altman is pure coincidental.
From yesterday you could run the dense qwen 3.8 on 1080ti. The research will continue.
Uh-huh.
Omnipotence is not expected, and also not required to kill all humans.
(NB: while I'm generally a follower of the Rightful Caliph, I'd disagree on one or two of the particulars there; FTL's about the only one where I'm more than a step or two away, though.)
Holy shit, has Yud finally updated to however small an extent? Lemme see...
Alright, that's a start...
Wow, that's actually more humble of him than I remember...
...of course, there it fucking is.
...yyeeaahh, business as usual. closes tab
I don't know, "superintelligence, ergo nanobot swarms in your bloodstream" seems pretty close to Yud's sincerely-held beliefs, if not his beliefs verbatim. He absolutely, straightforwardly believes that intelligence is the greatest force in the universe, and since intelligence ≈ power, then trivially superintelligence ≈ omnipotence for all intents and purposes. You can plainly see it even here, in the way the rest of his post gets instantly sidetracked into minutiae of human extinction as soon as it hits his favorite bugbear, nanotechnology. I'm surprised Ctrl+F "nanobot" has no hits here, I remember nanobot swarms getting frequent mentions in his lesswrong posts, has he finally wised up to not scare the hoes?
He didn't mention the word "nanobot" in the second linked article either. Probably has something to do with the fact that when people hear "nanobot" they think a mini-robot made out of metal, and that's not what he's talking about.
(I call the thing he's talking about "Life 2.0", and while I might not be quite as bullish on what can be wrung out of it, "kill the biosphere" with Life 2.0 is a DEFINITE YES because I know the skeleton of one method.)
Your "for all intents and purposes" is doing a lot of work. "Omnipotent", to me, means God/simulators; someone who has write access to reality. You can be very powerful without having that.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I call this magical thinking. The idea that AI will discover some magic that makes them easily superior to humans.
I don't say "magic" to crap on this idea. There is the famous quote ~ "any sufficiently advanced techno is indistinguishable from magic".
I think of IQ as leverage over physical forces. But undiscovered physics could amount to a cheat button.
I recall years ago on lesswrong people making this point. That some people are ascribing magical powers to AI when imagining they'd invent super technology very quickly or escape from their hardware by directly manipulating the computer components. I recall the response always being the Yudkowsky followers bowing out of the conversation. Explicitly saying they're bowing out. Which is a thought defeating cliche to prevent themselves from considering contrary ideas.
Not too be too harsh to them, but reciting that cliche to shut down discussion when getting any pushback against Yudkowsky's sillier ideas is a real anti-rational move. Stuff like that is why they had to speak in code avoiding the word "cult" to describe themselves. Asking users to please stop writing the word "cult" or search engines will associate them with cults.
I would appreciate receipts on this; there are alternate explanations which I'd need to rule out to believe your conclusion.
(AIUI, Yudkowsky is not a big fan of the "it's not about convincing your debating partner; it's about convincing the audience" school of thought so popular among SJ. Hence, if someone demonstrates that he is not amenable to reasonable discussion, there is no point continuing the conversation - and politeness dictates that one note this rather than simply ghosting. I can't distinguish between these hypotheses from the information you've given.)
I mean, the Ratsphere as a whole (as opposed to Leverage and the Zizians, and maybe the Vassarites) doesn't seem to meet any of the criteria for a destructive cult. Arguably it's a benign cult, but lol who cares.
More options
Context Copy link
More options
Context Copy link
What might you imagine undiscovered physics doing or accomplishing?
You are kinda asking what are the unknown unknowns.
Or at least the bigger question of how much of phyiscs have we uncovered?
I don't really know the answer. I will say that nuclear physics looks weird given everything you might be able to learn about the world. And nuclear physics gives us nuclear bombs.
Is there some force of nature we don't understand that is manipulable and offers large amounts of leverage? I don't know. That is the the question in my mind.
More options
Context Copy link
That's obviously unknowable, but you can look at what newly-discovered physics has done to get an idea. Going backwards through Nobel Winners, and filtering out any without obvious nearterm engineering applications, I see:
It's entirely possible that the next physics discoveries won't be very important, but they might be mid-range revolutions like the CCD and LEDs or era-defining like the transistor and integrated circuit.
There's also no need to stick with physics, such as with CRISPR in biology.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Anti-Yuddites Don’t understand that you don’t need omnipotence to do major damage here. AI is probably, if so directed, perfectly capable of taking down the banking system. And since few people have cash on hand and governments could not produce that much cash quickly, you could easily destroy the global economy that way. Most people would be ruined if their bank accounts were deleted, and im not convinced hard copies of that data even exist. You can probably easily assume the same of most infrastructure which isn’t up to 2000s standards of two factors or even strong passwords. So you could direct AI to take out power grids, water treatment plants, nuclear power plants, and traffic lights. That doesn’t require even much intelligence, but it certainly doesn’t require omnipotence either.
So the AI apocalypse can be averted by hiring two incels and making them type ip table rules here and there?
I agree with your assessment of the infrastructure openness, but why is everyone behaving as if this is immutable and unmovable state of the things. Securing the infrastructure is actually trivial.
If we start living in a situation in which the average ios and ps5 time from version release till jailbreak is <24h I will take the unstoppable hacker hypothesis seriously.
The most obvious threat - that AI could make a breakthrough in prime factoring or discrete logarithms - is somehow absent in the list of AI doomers.
OpenAI agents trivially get out of their "sandboxes". Opus 5 + a bit of human effort was used to hack OpenAI as recently as in July. This is the organization with IP ostensibly worth hundreds of billions and ≈infinite compute budget for automated pen testing. How, then, is this trivial for anybody else? People are really goofy. I see 1234 passwords all the time. Causing chaos at 300 tok/s is what's trivial.
OpenAI is, from the fact they recently got hacked and the fact they couldn't set up a proper sandbox, demonstrably not that good at information security
More options
Context Copy link
Do you believe that frontier lab / huggingface cybersecurity is more competent / advanced than most tier one banks, intelligence agencies, etc?
Most? No. I wouldn't be surprised if they were mid-pack. This isn't to say I think OpenAI is really good.
More options
Context Copy link
More options
Context Copy link
You can't protect against stupid. I would love to see an OpenAI agent escape from sandbox I have set up. OpenAI don't give a fuck about opsec. You don't need agi or asi or even LLM to hack stupid.
It is as if when faced with Y2K we said - we don't have to worry about the old code, we need to find a way to stop the arrow of time.
If we have to be worried about current infra - then harden it. It is not hard. Pun intented. Just putting a small rule in the firewalls from which ips the network is allowed to receive connects, or creating intranets would go a long way. Nuclear powerplant doesn't need to accept connections from all of ip4 and 6 spaces.
If we have to be worried about super intelligence that will find unending stream of zero days bugs in the logic, silicone - then air gap it. But it is not an argument for slowing down.
So far we haven't seen new android roots, new iphone and ps5 jailbreaks. We know how to make stuff more secure.
More options
Context Copy link
More options
Context Copy link
Due to the sorts of inequalities that people complain about regularly, banks have a lot more resources at their disposal than hackers typically do. Plausibly, they can afford Fable and Astra tokens to pentest and harden their own networks ahead of script kiddies having the means to buy them. I wouldn't expect that to last forever, but if they put effort into it they might not lose. Nation-state actors are different, but even then Western nations can afford better models generally.
Not a strong hypothesis, but pithily "the only thing that can stop a black hat with an LLM is a white hat with a better LLM."
Yeah, I'm actually fairly optimistic about hacking. If everyone has access to roughly equally strong AI, then what really matters is whether hacking fundamentally favours defense or offense. And if we're talking about a single important target like a bank, it seems almost tautological that the bank has an edge on defense, because it is at least as hard to find and exploit a vulnerability as it is to patch it. You probably need an order of magnitude more effort to hack a bank than it needs to secure itself.
Now, the kind of hacking where you just cast a wide net and take advantage of all the weakest links (e.g. making a botnet, or stealing user data from a random incompetent company) favours offense, but that doesn't strike me as quite as worrisome.
Bioweapons are the main threat where defense seems far more costly than offense, which is why it's what people are (correctly) most worried about.
Bioweapons are most dangerous for their wannabe creators. If the Houthis decide to create their own AI enabled bioweapon program it will lead to depopulated Yemen.
Agreed. The risk still exists that some group would want bioweapons anyways. Aum Shinrikyo would have made and deployed bioweapons if it was feasible. And indeed they'd be at least as likely to be killed by the weapon as their victims. That's little comfort to me.
More options
Context Copy link
And, what, the virus will just stop at the border because it doesn't have a passport? In our modern highly-connected world, a contagious virus is almost impossible to contain. China and Australia did arguably manage to succeed with COVID, but it cost them dearly.
This is why nukes, and not bioweapons, have always been the WMD of choice for state-level actors (and how we got a basically unprecedented level of global cooperation for eliminating smallpox). A weapon that kills 50% of your enemy's population and yours just isn't all that useful. But if the bar for developing one is lowered, then a small terrorist group might decide that depopulating the world would be just spiffy.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Most important software that implements cryptographic primitives that rely on the hardness of these specific problems (prime factoring & discrete logs) is already in the process of being deprecated / replaced, just because it's known they'll no longer be secure if / when quantum computing hardware scales.
I agree with the major point, though: it's not unlikely that robustly super-human mathematics will destroy many of our cryptographic protocols in the near [1] future (not just particular software implementations of these protocols, but the protocols themselves). At this point, I would not be shocked to see a SHA-512 collision published (probably on Twitter).
[1] I don't know what "near" means exactly.
More options
Context Copy link
More options
Context Copy link
Social Security is also probably capable of taking down the banking system. "Taking down the banking system" is very clearly a human-scale problem with human-scale solutions; Yuddism is entirely about appealing to supra-human-scale problems that purportedly don't have solutions outside Yuddism. I am 100% on board with the argument that AI safety, were it to be a thing, should focus on the human-scale problems and not the supra-human ones, which I'm not confident exist and which I'm not confident have reachable solutions if they do exist. But this approach is very clearly incompatible with Yuddism.
More options
Context Copy link
More options
Context Copy link
Eh, in the sense that the AI can do independent target discrimination, sure, but that's already a solved problem (LRASM can do this, for instance) although it can be iterated out further.
World militaries and terrorist groups have no interest in making sure their drone processor can write poetry, or whatever.
I'm sure at least someone has considered more difficult target discrimination problems, which seem to edge closer to the "poetry" use case. "Is this language Russian or Ukrainian", or discerning between common Eastern Bloc hardware used on both sides seems relevant and difficult, especially with active countermeasures on silhouettes and shapes going on. That meme with three fingers seems relevant.
I just don't think sticking Astra or Fable (as opposed to a lightweight image processor) is the solution here.
The problem with a lightweight tool is that it can be fooled by changing small details (e.g. using Ukrainian spelling on your Russian tank). But those are exactly the same sorts of things that are going to trip up Astra, too, because it is going to be instructed to use a similar decision matrix, which, again - why not just use a lightweight image processor if you are going to give them both the same targeting library.
In many ways, I think militaries would prefer a tool that isn't doing independent reasoning to strike a target. The reason for this is pretty obvious: independent reasoning historically leads to friendly fire. At least if your independent reasoner is on the bridge of a ship (and this goes for a human or a larger AI model) and it screws up, you can figure out what happened by conducting interviews and data-dumps afterwards.
If you design a missile with an independently reasoning seeker-head and it starts friendly-firing, you may have a hard time figuring out what is going on because there may be nothing left of the
malfunctioning"misaligned" system to evaluate and its reasoning may be too complex to review in a timely manner. So if something happens, you might have to yoink an entire production line of weapons and/or re-train your model in the middle of a war. If a deterministic program is friendly-firing, it means:All of which are probably easier to figure out and patch during an active conflict than "what is going on with Astra" - witness the difficulties the AI companies are having following chain-of-thought now, when they have total access and control over the models.
I am not saying that we will never get the fabled "self-aware weapons system" from AI, and I wouldn't be surprised if testing is done relatively soon. But in the current threat environment, it strikes me as buying a sledgehammer to kill a fly.
That's not to say that LLMs have no military value at all. I just don't think they are optimized for weapons guidance.
Those sorts of targeting concerns are [ETA not] strictly new: there are plenty of WWII accounts of poorly-designed torpedos swimming in circles back toward their launchers. But they are valid concerns, and there are better IFF techniques (not AI) available for that sort of thing.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I can only assume that you haven't of the Vogons.
I am in no way specialist on how paperclip maximizers work, but to me the actual capabilities of the models are mostly dressing over the underlying tech. And the fact that they use the inefficient human language is mostly artefact of the google babelfish experiments.
To make drones properly lethal you need meshed swarm coordination, ability to create and adjust tactics on the fly, without much centralization in conditions of severe jamming. So they do have incentive to cram as big models as possible in as limited hardware.
Surely that sort of thing is in the domain of the CIA!
You're sort of missing my point - the goal here is to cram the most optimized models - that is, the models that are going to best allow them to do meshed swarm coordination, or whatever. Larger is not always better; better is better.
I have no particular reason to think that the direction LLMs research is currently developing is good at all for that sort of thing. Cramming Astra into a drone is going to result in something that's much slower and less decisive than a simple deterministic script, because LLMs are compute-hungry. That's not to say that neural networks won't be helpful - computer vision is great, for instance.
But LLMs are not the only form of artificial intelligence, and for most military tasks a strong general-purpose LLM is both overengineered and poorly suited for the mission, especially the mission of terminal target determination. For intelligence synthesis, they likely have some promise.
You have the right of it.
The additional issue is data, to a shock of nobody paying attention. Military data is near non-existent. And the data that does exist is low-resolution, or tightly controlled by non-ML/AI folks who have been trained to have negative desire to share (The Security Clearance process over time selects for a type). LLMs are the Ford F150 of ML models. Half the battle is often getting a very specific tank configuration that is actively trying to hide in foliage from the very CV models you are trying to detect it with. You have N=3 samples of this tank sitting on a tarmac somewhere and it turns out the ability of your model to extrapolate that picture to 3.0 GSD satellite or drone footage is abysmal. And that is before it even starts attempting adversarial countermeasures.
There are of course solutions, but they don't work as well as folks imagine they do, including generative image generation. Most of the examples of drone based hits are FPVs or FPVs with a "wait" situation where the drone hovers, detects a general class and is then given approval by a human analyst who manually reviews the vid/image.
Very interesting! I actually thought that open-source imagery would be better resourced than that (although of course it's pretty easy to mess with the profile of a tank with a can of spray paint or a camouflage mesh, which I guess is part of the problem).
Always very heartening to have an industry professional tell me I haven't gone completely off the rails, thanks.
There are what a couple hundred grainy low res videos of Ukr drones targeting Russian tanks + other vehicles? It's well below the threshold to train an ATR model on data quantity, without even getting into quality. Stock images are from non-operational angles (you for the most part aren't level with targets, its an isometric or overhead view, NERF to solve that is expensive and has dubious performance improvements, and having worked in the generation of low-sample military images, it can actually be quite hard to secure exemplars in EO imagery. SAR imagery just sucks, speckle is multiplicative noise which makes it nearly impossible to remove and hard to detect around.
Humans are obviously much easier to target, however then we get into friendly fire and the fairly disturbing optics of AI drones targeting your own soldiers... There are logistic issues around distributing RFID and similar tags to prevent it, that to my knowledge are being worked through. But the main goal of drone warfare is destroying expensive vehicles with cheap drones.
Training datasets for the ATR stuff are being developed but its sort of an incumbent's advantage, with active known defense contractors having most of the customer connections to be able to get the data, and the resources to label it. Unfortunately they are also slow AF, which is why defense tech startups sorta eat their lunch in certain spaces. Transfer learning is something I've seen recently, taking civilian jeeps with SAM-esque models and applying them to humvees etc. It's always much easier to train on your own nations hardware anyway. Though it gives bad optics to Brass, who don't like it, and an open question on whether it will transfer.
For Russian (or American) kit, I figured you would have good most-angle photos from parades, demos, and such. You're right though that these are mostly taken from the human level.
I imagine that another obvious issue distributing RFID tags to your troops is that they can be used by both sides. I realize that in theory they will only respond to coded radio signals that in theory only one side can transmit, but if you're putting them everywhere and then putting the transponder in something as failure prone as a kill-drone your enemy's going to be reverse-engineering your IFF on like day 2 of the war.
I've speculated, based on my knowledge of machine imaging, that we'll see "wartime camo" become a thing in future wars, with "peacetime camo" designed to be something maximally different from operational paint schemes to confuse ATR. Apparently the Russians are already using a variety of disruptive paint schemes in Ukraine. We can only hope it results in making razzle-dazzle great again!
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Wow my field has made it onto TheMotte, spooky. It's not quite so simple as big LLM models on limited hardware. There are a lot of networking problems on swarm interfacing if you want something that is truly adaptable. A lot of swarm robotics is essentially "dumb" control algorithms with software permutations, because LLM-agent hallucinations are rough + LLM sizes require truly monstrous hardware on drones. There's work on fitting them to Jetson Orin Nanos' but it's a slog just from the amount of compute needed + performance of smaller quantized models etc.
This is generally a case against the LLM + Drone combination, too much extra stuff not needed. Part of the problem with current AI research is that it is all in on LLMs but the branch off of not-LLMs but with LLM-like capabilities for reasoning for tactics or strategy are not there. If you want the reasoning you get poetry on your drone too. It actually goes beyond mesh networks as well, as a mesh network is every node connected to every node. You actually want localized connection networks that do multi-hop message passing. Much more resistant to jamming.
Idk maybe I should write an effort over-post on drone swarms, I love this subject.
raises paw Interested!
More options
Context Copy link
I'd love to read it, if you ever do.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I find myself frustrated at a lack of what I expect would be cool theoretical results on bounds for intelligence. It seems clear that tiny microcontrollers are never running meaningful models, but I'd have expected some "this much hardware/weight count for this much intelligence" bound. But I suppose we still don't have a raw measurement (maybe even definition) of intelligence.
My intuition is that my (not-web-enabled) toaster is never going to achieve much useful thinking, and that actual intelligence has diminishing returns with scale in ways that make "super intelligence" useful, but not unboundedly powerful. But I don't have a strong argument for that.
Pithily: Modern AI has a Claude, but lacks a Claude Shannon.
You can pack a surprising amount of intelligence into a small neural network. There's the demo floating around of the fly neural network that's been trained to solve a Rubik's cube. Presumably it won't be able to solve a Millennium Problem, but then again, who knows.
You may be interested in Hutter and Shane Legg's program (AIXI etc), which seems like a pretty appealing definition of intelligence, with the minor downside of quickly becoming incomputable for anything but the simplest toy problems.
It won't, it's been trained explicitly to solve rubick's cubes in a way that is unlikely to generalize past rubick's cubes. This method of small network + RL on some straightforward game problem has been around forever. I remember training an RL model to play Euchre back in the day through self play. You can fit a lot of performance onto a small model that has been trained in a supervised manner.
Oh, of course. I mean more generally: shorn of everything else, does a fly NN have the capacity to e.g. write Lean proofs? As you point out, you can fit a lot of performance in there. I don't think it actually can, particularly if we're talking about novel, interesting theorems. But we don't have the theoretical bounds to say that with certainty.
Can the fruit fly grok, say, the axiom of choice? It's both a simple and a very abstract concept. I'm not sure, but it's an interesting question.
Highly speculative:
I guess it depends on what you mean. It wouldn't be surprising to me if there is some fruit fly neural configuration that could consume and produce some specialized proof language, and even produce (with enough compute) something like well-ordering implies choice and vice versa. You can fit a type checker for dependent types in a hundred lines of code.
Not many people would call that grokking, though. Actual learnability and intelligence need far more dimensionality; that minimalistic net above would have to be engineered or compiled, not learned. And even given the prover embedded in its weights, the fly would never think about well-ordering or be able to identify it as interesting. That comes from groundedness in real world experience, and creating a generalization of its real world experience that can be mapped to the mathematical abstractions it knows about (which, I hypothesize, requires the kind of dimensionality that the human brain has).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
How does Qwen 3.8 Flash perform? What kind of K/V cache are we talking about here, on consumer hardware?
I'm an AI fan but I have issues with even the biggest and strongest models for my usecases, which ironically enough is AI development (non-cheating AI in a strategy game, that is). They get there, we are making progress - but with no small amount of fumbling along the way. Testing various ideas and strategies takes time. Qwen 3.8 Flash is below Qwen 3.8 Max, which itself is below Kimi K3, right? And that's below Astra and Fable. And my usecase is nowhere near ASI development.
I agree that AI will improve in cost-efficiency but consumer hardware seems like a stretch.
Anything fast or nimble is going to roar and bellow from datacentre-grade compute if it can whisper on consumer hardware.
I can run Qwen-3.8-Flash (120B) at Q4, K/V cache, on an nVidia 3090 and a Core i5-14400 (albeit with a lot of now-expensive RAM), with simple llama.cpp run, around 5-2 t/s at 100k available 16-bit kv cache. Dropping the kv cache to 50k nearly doubles performance. Qwen-3.8 in general defaults to a heavy thinker, so that's worse than it sounds -- a moderately complex problem can burn 30k tokens -- but it's the sort of thing you can leave crunching on a problem for a while and be happy about the answer.
(Comparisons: Qwen-3.8-27B runs about ten times the speed, and Gemma4-26B runs basically faster than I can read it.)
For intelligence and capabilities, the comparison to frontier stuff is rough. Low-parameter models just don't have some information, and with either hallucinate or just nope out, no matter how well it had to be present in the training data. Indeed there's been some efforts to trim low-value knowledge from public models to optimize them for specific use cases, with weird results.
And home users have some rough spots. Both quantization and abliteration drive perplexity and errors, and the harnesses to find and debug them live aren't well-established in the open source (or free-as-in-beer) world. It's fascinating to read a logic trace that goes into surprising depth, but it doesn't do much if the program output doesn't work. The errors are small and embarrassingly simple for a programmer familiar with common JS errors, or for other models to catch, but non-programmers would likely struggle to explain what was even going wrong.
((Also note: the game's not good or fun, even when it does 'work'. That should be expected given the lack of specificity, lack of agent harness, or even a real iterative process, but it's also something no human would do this way even as the core idea it came up with is kinda clever.))
That said, intelligence can be surprising. If you want a model that can make connections between input tokens or parse through mounds of data, you can get away with stuff much smaller and more energy-efficient than you would expect. I would not, a year ago, have expected you could get spatial reasoning worth spit in a 27B model. A real big curveball isn't the most likely thing, and I wouldn't put a ton of money on specifically Jev doing anything ridiculous, but I wouldn't bet against someone coming out with a two-fold performance or intelligence improvement for inference in this model class before the end of the year, either.
More options
Context Copy link
More options
Context Copy link
How does that work? That gpu has, what, 11 gb vram?
https://prismml.com/news/bonsai-2-27b
It is with small context and 20t/s , but 24/7 are a lot of tokens.
This is a dangerous type of company. They are (presumably) thinking they are engaged in well-doing by democratizing intelligence or something like that. But soon the worst scum of the world will have very powerful and uncensored, non-refusing models running on cheap hardware and probably smartphones too, in not too distant future. This model you've linked can even be run in a browser. No know-how required at all.
Yeah, imagine having a model that does what you tell it, how horrible.
Did you know that this existed in other forms called "books" for the last hundred years? And what was the maximum we actually got in terms of terrorism? Arguably the scariest it got was Aum Shinrikyo, who had a bunch of people intelligent enough, and driven enough, to wage chemical and (if they actually had the materials) nuclear civil war. The secrets behind those mechanisms were at the time widely-published; everyone knew how the Hiroshima device (and sarin gas) worked maybe 20 years after it was used.
The only really dangerous thing I could think of would be biology, and perhaps after the inevitable novel biological attack (which will likely be least-effort) we'll see agar become a controlled substance. But the models aren't ready for that yet and it still requires a researcher with their head on straight, which is selected against when we look at the average (for example) mass shooter or mass car-driver; even Aum's highly sophisticated attack in one of the most crowded places in the world only managed half the death toll of an angry retard attacking an office with a gallon of gas).
Then again, I kind of believe in the American way where "having every tool necessary to destroy your neighbor should they piss you off, and every day of your existence is a choice to deal within the rules" is probably more stable than highly authoritarian regimes simply because the existence of the veto helps ensure those rules remain fair. If the State is otherwise capable of complete totalitarian control, which with the coming tools they will, then individuals under it must have some power to check it or there's no reason, from a realpolitik perspective, not to go full 1984 and enslave them.
Even then, political violence (which the more intelligent types prefer) tends to be targeted; the prototypical example perhaps being an autonomous turret set up in northern Ireland trained on pictures of Oranges, set up as non-lethal simply to make a point, or otherwise depending on how angry they are.
More options
Context Copy link
What do you mean soon? You can already run completely uncensored, non-refusing models on regular consumer hardware. Huggingface is chock full of heretic uncensored model weights.
You need decent consumer hardware/vram to run the good models. Soon the models will be much smaller, and thus much more accessible to poors.
Even I, with a 7 years old second hand laptop can run surprisingly decent models. Any gamer with midrange GPU can run Qwen 35B A3B models at a decent speed (~20t/s or better) without having to have any particular technical skill beyond knowing the magic word "heretic" and doing a huggingface search for it. Anyone with a 24 GB GPU can run uncensored Qwen 3.8 even without any "dangerous company".
Anyone "dangerous" already has full access to those models right now if they only so desire.
Oh god. Pandora's Box is already open. hides under bed
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I already like them. No need to sell them to me harder.
Oh. Cool. How will you and yours prevail in the Cyberpunk 2077 world of tomorrow?
As a corpo I'll do well. Be overworked and scared of terrorists perhaps, but materialy wealthy.
More options
Context Copy link
By being smarter, more ruthless and more inventive. What your ilk don't understand is that Cyberpunk 2077 is the good future awaiting humanity. Leaving AI under government control means 1984 ... at best.
Cyberpunk 2077 is also the same bad future, where "the net", technology and AI are also under the control of the same corpos which have morphed into a fusion of corporation and state. Indeed some of the quests revolve around said corpos being too greedy and arogant in their use of AI as a WMD and losing control of such. Several times if you consider Johnny to be a fusion of an engram and ai.
More options
Context Copy link
I don't even disagree, but you are aware of that whole the-AIs-are-about-to-break-through-the-blackwall thing happening in the background of C2077?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The mood is already turning against AI even while the labor market is strong, it’s hard to believe that won’t get much worse as large scale layoffs start to bite. As we know, corporations exist primarily to serve their employees rather than shareholders or customers, so (as @RandomRanger says below) this is more likely to be in the ‘the guy next door opens a factory with a tenth of the workers and half the prices’ form of creative destruction. That will force widespread changes in the labor market absent extreme political intervention1 , which necessitates politics, and could happen in a matter of months (or not). I also think it’s clear by now that a Chernobyl or at least Three Mile Island (although I expect the former) level casualty event is imminent, at least from my dilettante’s understanding of the HF attack, which will only make the backlash stronger.
The Chinese, of course, have much less of a social safety net than any Western country, a hypercompetitive entrepreneurial culture used to driving down margins, and a competition between provinces for subsidies that essentially encourages lossmaking businesses. All this means that even if CCP directives ban layoffs (and they will), that would not contain a labor crisis in most sectors. Inertia can only take you so far.
I don't think "large scale layoffs" is necessarily the only future here. I'll certainly admit it's possible, but historically automation hasn't always gone that way. A century ago, we had lots of accountants, and most of their time was spent manually balancing arithmetic, maybe with a mechanical adding machine. Along came the personal computer (earlier for large businesses with mainframes), and most of that work can be automated pretty trivially. Yet there are more accountants today than there were a century ago. What changed is that they're not doing arithmetic, and the complexity of what they do has grown substantially. For all the complaints about the growth of the tax code (and frequent related stunts by politicians), we are able to handle that complexity because we can offload the line-by-line balancing to software. Some of that complexity is probably wasteful, but some of it is also better auditing and accountability, or targeted governmental incentives — some, again, is probably wasteful, but some seems reasonable.
I don't have quite as coherent an argument (still a work in progress), but it's also the case that "recession caused by mass AI-driven unemployment" is actually a self-own by corporate AI overlords in limiting their upside. If you replaced half of human workers tomorrow at a cost savings, any not-truly-essential business is going to lose customers who can't afford their wares: tech companies fit pretty high on Maslow's pyramid, and most of the base ones are long-commodified — at one point, most of us were farmers, then along came the tractor. Nobody in the West is starving because they can't get a farmhand job. The cratering of farm employment is, in fact, why I get to work an office job, rather than out in the fields somewhere, and I'm not sure I'm worse-off for it.
But that latter part is, again, a half-formed argument.
The problem is that essentially there’s little chance that there’s a limit to the complexity that AI can handle. It’s not just a calculator, it can do all kinds of data analytics and so on. That’s the issue of saying that “maybe the people will just do something else” is that we’re talking about a general intelligence that can learn to do the next jobs created — probably before you can set up a training program to teach a human to do that job. It takes 4 years of school to make an accountant. It’s probably a lot less time to give the LLM training data and have it learn to do those tasks. And that’s true for a lot of tasks. Actuarial skills are basically “find patterns in data”. X-ray and diagnostic imaging is the same thing, but with images. So we’re going to have a problem simply because any newer, more complex job enabled by people using AI can be done by AI probably before you can teach a human to do them. Then that enables more work, except that AI can do that work too, and probably the next set to infinity.
Eh, Assuming we don't EMP ourselves into the stone age (HV transformers have replacement times measured in months) we would simply use government FIAT to mandate some % of human employment or simply forbid AI use in certain sectors.
Two problems:
1). Unless you can do this enforced at a global level, it doesn’t matter (the same is true of the safety stuff btw) because if it’s illegal to use AI to do say translation work in the USA, then the cheap solution is to simply use AI to translate things in Canada. The humans are still going to lose those jobs, you just get the added bonus of not only losing the human paychecks, but the taxes paid by the company as well. Or if we’re talking safety, we slow down and limit our technology for safety. China is under no such obligation and will happily lap the safety conscious American companies and dominate AI.
Second problem being that effectively, there is no way to enforce such a treaty even if we make it. The UN is a glorified debate society. To the degree it allows wars or issues arrest warrants, it’s basically dependent on member countries to go along. It doesn’t have a military or police force, it doesn’t have a police force. It doesn’t have the legal right to put boots on the ground to arrest war criminals and instead must depend on the countries to volunteer to abide by the treaty or resolution. If Trump were accused of war crimes and was supposed to stand trial in The Hague, so what? The USA is not going to agree to let a sitting president be tried in an international criminal court, no military is going to volunteer for the suicide mission of trying to defeat the USA military to arrest him. And other treaties are on the same honor system. We have the Paris Treaty on Climate Change, right. But countries face no actual penalties for not keeping to their carbon limits. That’s the same international body that’s supposed to police the AI treaties when countries have trillions on the line if they win the race. The first to create a human level AGI has enormous power and influence and can basically print money. It’s worth trillions, conservatively in GDP. It means military superiority. No one is going to even pretend to follow the treaty if it even exists. The only use of that treaty is to try to trick other countries into slowing down so you can catch up or pull ahead.
I'll give you that, but this assumes some Theoretical Super inteligence type AGI deal, we could for a long while hover over mundane human tier AGI, still something more than we currently have with the LLMs, but nothing that's not already in play (billions of general intelligences, running about, eating burgers, giving titjobs etc)
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Of the job categories listed in the 1960 US Census, supposedly only one has disappeared completely: elevator operator.
Nevertheless, I think the complexity argument might fail for accountants. Complexity actually favors AI. They are good at it.
When you talk to rich people, everyone is looking for an accountant. They all think that their accountant is bad and want a better one. I've only had bad ones personally. Now I just do things myself because I don't want to deal with errors. AI has been a godsend. I think it's likely that AI will do my taxes in April 2028.
But there is a silver lining for accountants: knowing which rules to bend or break. If it treats taxes like it treats code, AI will happily fill out a 500 page form so you can pay $0.04 to the state of Arkansas for some K1 you got from an oil services company. The accountant knows to throw it in the trash. They can tell you to ignore that letter from IRS because it doesn't get serious until the nth notice. They know how to ignore the rules that contradict the other rules. They know that certain tax dodges are okay because all their clients do them and never get audited.
I think it's somewhat similar to driving. The system only works because people know when to break the rules. Will AI break the rules? If so, will they be liable for that? Humans might stay in the loop just so they can break/bend the rules.
"Computer" is also no longer a job category thanks primarily to the machine named after said profession.
That one was gone before 1960.
More options
Context Copy link
More options
Context Copy link
Hilariously I've been in elevators with an operator.
I'm pretty sure another silver lining for accountants is that "my freakin' AI did it wrong" is probably not an excuse that the IRS will accept. They will accept "my accountant did it wrong."
Was this in a heavily Jewish area, perchance?
New York City, so yes.
I think elevator operators are standard in tall buildings which are themselves tourist attractions like the Empire State Building, the Shard, the Eiffel Tower etc.
Of course their real job is mostly guest herding, but they do operate the elevator.
I think its more about the religious prohibition on doing work during the Shabbat and the interpretation that pushing a button on a machine is work, so they have people to push the button for you. Hilarious and archaic at the same time.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Definitely. It’s the same problem that self driving has where every Tesla that crashes with FSD is nationwide news even though it’s 10x (or more?) safer than average drivers.
Those things only apply on average random samples. As soon as they hit a consistent failure mode, you end up with Problems.
More options
Context Copy link
I think it's worse. In the case I am familiar with, (sample size of two) as far as I can tell, when a guy did something on his own, he did okay. When he tried to use AI to do about the same thing, SNAP!
So either the AI screwed up big time 50% of the time, or 100% of the time. Either way, not great, and seems worse "per task" than self-driving cars.
Weird. This is so far from my experience that I wonder what you guys are even doing? Like, you only tried AI twice?
If you’re a senior dev, and AI is not banned by your org, and you’re not personally using AI, that seems very inefficient.
Oh no, I've tried AI lots more than twice. Definitely witnessed more than one hallucination.
I'm not in software. The mistake I'm talking about wasn't made by me, so I don't have any insight into how it was made or how often the user was using it, or what model.
To give you the sort of idea of what happened without doxxing myself, imagine someone turned in their PhD dissertation and it (or parts of it) turned out to be AI written, and the AI had made stuff up.
Was he using a "cutting edge" model? No idea. I've had Opus goof stuff up for me before, so I don't necessarily trust even the higher-end models not to make mistakes from time to time.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
AI fails for accounting not because of complexity but because of care. Accountants have to get the numbers right, and occasionally hallucinating shit isn't going to cut it. Maybe it would work for mob accounting, but I suspect the mob wants their made-up shit to be less random.
I doubt it - the IRS don't start with your fingernails when you make an error.
Correct, the IRS traditionally starts by hanging you upside down and shaking you until the money comes out of your pockets. Then they start on your toenails.
More options
Context Copy link
More options
Context Copy link
If you have three independent AIs do the numbers and they match, they’re likely right.
More options
Context Copy link
The hallucination argument is losing power. Harnesses will fix hallucination much like they have for coding.
I'm not going to straight up ask Astra to do my taxes, but I'm fully confident that Astra+harness is capable. So now we just wait for the harness. No doubt lots of smart people are working on it.
Accountants don't get the numbers right anyway which is why I do my own taxes now. It's a boring, frustrating job and attracts people who like money more than having interesting work with predictable effects.
Harnesses fixing hallucinations is news to me. I just spent an hour today putting a Junior's head on straight after codex sent them down the garden path for a day and a half.
It's not really about "harnesses" as some dedicated tooling, it's about meta-instructions and models generally becoming good at following instructions over long horizons. Like, I just wrote in my AGENTS.md some stuff to the effect of "periodically spawn subagents for independent blind audits, create teams with different scopes and domain-customized personas, check with web access, sometimes step back, iterate until you get clean results", and even the latest DeepSeek will diligently execute all that and massively improve its reliability. Astra Ultra won't even need such handholding. They'll still make mistakes, rabbithole into irrelevant minutiae, perform suboptimally on cost, but for labs this is all easy to improve with More RL, and humans are flawed too. I am sure that a junior's level can be automated today.
And yet.
ChatGPT Sol absolutely.
insists on
writing.
Like this.
In a shitty.
broken up.
style.
If I ask it to demonstrate a writing concept or somesuch and there are people in the scene. It will happily tell what the problem is and how to supposedly prompt it away but will then keep on writing like that. So much for "good at following instructions".
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
No, it most definitely is not.
Do you use these tools on a regular basis? My experience with hallucinations is that they are extremely reduced.
Same. I can't recall seeing one with Codex and 5.6-Sol.
More options
Context Copy link
It is still a problem with whatever models people are using. Just had to deal with a pretty bad incident at work a couple of weeks ago (thankfully not one due to any failures on our part).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I agree, but it doesn't seem like there's been an emergence of an Official Progressive Position on AI. Not like there is on Israel; or on Elon Musk; or on gay marriage. With respect to AI, I haven't noticed an army of NPCs uniformly and uncritically repeating the same talking points and buzzwords. I haven't seen much in the way of virtue-signalling competitions. So it doesn't (yet) feel like a culture war issue to me.
It's interesting to note that in China, there are large numbers of State-Owned Enterprises which really do (at least in practice) exist to provide jobs and effectuate state policy. From what I understand, a common scenario will be that a State-Owned Enterprise will have a publicly traded subsidiary and the shares of that subsidiary trade for significantly less than the scrap value of that company. Because the parent company won't allow the subsidiary to just be sold. In other words, it's known and accepted that these companies really don't serve their shareholders.
So I would expect that if there is a worldwide AI boom, the CCP will make use of these State-Owned Enterprises to stabilize their economies and employment figures. After all, they are doing that already.
I've been seeing Hinge profiles with demands that anyone who wants to fuck them not use AI, at slightly less frequency than the usual ACAB/Palestine. And I've heard the data-centers-use-wAtEr rant from my local frothing leftist co-worker once or thrice.
Are they referring to general AI use or use of AI in their dating profile, messages, pictures, etc?
There's a really strong movement against even knowingly consuming products made using it, although it's not quite as straightforward as right/left or red-tribe/blue-tribe. The furry version isn't much of a surprise, but it's definitely something with a faction in normal (or at least as anyone on modern social media can be normal) space, treating its use as a mental contaminant.
The furry version is understandable, that's a gooner culture where larping as a ware-dolphin-lizardkin with neon hair is part of the dream. And a significant portion of the culture ritual is consuming and exchanging 2d/3d and text fiction depicting such creatures/characters. In essence, "there is no soul in these coochie pics"
More options
Context Copy link
That movement seems to be a fairly straightforward job protection campaign among creative professionals (who are disproportionately leftist).
More options
Context Copy link
Basically I'll accept that it's a culture war issue when AI researchers giving talks on progressive college campuses get disrupted just like conservative speakers get disrupted.
More options
Context Copy link
More options
Context Copy link
I assume general AI use. Hinge is so cringe, conformist, and buzzword-y already, I don't see how AI could even be noticed there.
Assuming that's true, I agree it's evidence that AI is becoming a culture war issue. That being said, I haven't yet heard stories of AI researchers having their talks disrupted as would happen with conservative speakers on college campuses.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
It seems to me like there is. Generative AI is evil because 1) it's used by capitalists to save money by replacing workers and 2) it specifically replaces creative people. Of course, #2 is partly an artifact of the fact that it's creative people who have access to left-wing media in the first place.
There's no progressive position about AI taking over the world, but there doesn't need to be.
Well here's a thought experiment:
Suppose you are on a progressive college campus. Which is more likely to provoke a negative reaction from your progressive classmates:
(1) You are known to work for an organization the primary purpose of which is to oppose gay marriage;
(2) You are known to work for an organization the primary purpose of which is to oppose affirmative action and other forms of discrimination against white people; or
(3) You are known to work for an organization which is primarily developing improved LLMs.
It seems pretty clear to me that the first 2 are going to provoke a much more negative reaction.
Or to put it another way, in progressive circles, people are afraid to even suggest the possibility that the police are not in fact biased against blacks; or that trans rights are being abused by fetishists; etc. But if you argue that generative AI will lead to a wonderful future? Not so much.
I don't have close enough knowledge of a progressive college campus that I could tell you if saying that you work for an organization developing LLMs will get you attacked. But it doesn't seem implausible.
Well, I have heard lots and lots of stories of speakers who were conservative, pro-Israel, etc. coming to college campuses and having their events disrupted. I've never heard of this happening with AI researchers. So I'm just not (yet) seeing this as a culture war issue.
They get booed and jeered
I could see this leading somewhere, but I don't think we are there yet. I was talking about a situation where, for example, Ben Shapiro is hosted on a campus. Not as a graduation speaker but just at some club event. In that case, if it's a progressive-dominated campus, hostile people (who wouldn't otherwise attend) can be expected to seek out and disrupt the event.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Which ones want "an uncensored local model on every device, every man a centaur"? d/accs?
Me. And ironically, Gwern, despite his broad doomerism.
More options
Context Copy link
More options
Context Copy link
Somebody should hurry up and make an actual political compass test about this. I have no clue where I stand ideologically on this memeplex. Someone in the know give me a placement:
I think AGI will happen eventually, not sure I believe a singularity will happen in my life time. It's will definitely change society, in some ways for the better in some ways for the worse. Hopefully it will force humanity out of the cozy evolutionary saddle point we've gotten stuck in by being the dominant species on the planet with only ourselves to contend with. Centralization of AGI capabilities will only ever lead to authoritarianism, democratizing it will probably be the best for human flourishing in the long term. Despite being an overly cynical, skeptical, and pessimistic person, I mostly have rare optimism towards the results of AGI realization, on some level this is humanities child. I think that's pretty cool.
What ever happened to Guillaume Verdon? I'm not on Twitter, so is he still getting his 15 mins of fame, as a durable celebrity or has he fallen off as another startup hypemonger looking to "transform the world" by transforming hyper/notoriety into dollars for his startup, and consequently his own wallet?
Guillaume keeps hyping up Extropic, however his issue is that LLMs are in fact very good even on existing hardware (which is itself quickly improving), and his hardware is not general-purpose. Uncharitably, his startup has vaporware tech that could never work in a meaningful sense.
I tend to over index on skepticism around 90% of AI-related startups so even just the hint that this might be the case is enough for me to consign it into the startup-grifter bin. I've seen too much in how the sausage is made around converting fame and name recognition to financial profit in the tech space that I have even become cynical of actual technological innovations that reek of the same odor.
How is it actually done? What are the mechanisms?
More options
Context Copy link
From knowing him a bit, I believe that he's genuinely scientifically competent, a sincere believer in using quantum/thermodynamic effects for computation (to the point of developing a minor psychosis about the entire thermodynamic God thing), and prone to wishful thinking about its commercial viability. The pure grift component is modest (the current version of his hardware has no use case, and his real Big Boy Thermodynamic Chip can't be produced with manufacturing base available to him), but with the standards of evidence asked of small startups, the line is blurred by default.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think that even AI good enough to replace workers has minimal actual impact because most white collar companies do not try very hard to make money. Companies are run mainly as a lifestyle and socialization environment. Making money is a secondary goal. Making more money might mean higher bonuses. But the company as an institution does not have the planning or will to appreciate that and deliberately work to raise productivity.
They would much rather remain ignorant of the simplest AI applications than learn anything new or do anything new. If they do anything new, they'll do it wrong. They'll get GPT5.4 from Harvey or through Copilot or whatever as their AI solution. They have no time for doing anything new. Software engineering is an exception, people there are closer to the AI world and have more of a clue. Outside software it's a madhouse (exceptions exist but I'm talking about the general population).
The average white collar workplace doesn't particularly want to run more efficiently. Some of the biggest companies are run even worse. Google is run in an extremely inefficient way. Interacting with Google the company, or even just trying to use some of their products like Google Vertex is extremely difficult for no reason. Google acts in this needy, schizophrenic, loathe-to-make-decisions way, they stress out people dealing with them.
Even if Gemini 5 is smarter than the average Google worker, better in all respects, Google won't change significantly unless Gemini is working on projects at AI-speeds autonomously. Put in more intelligence into the dysfunction machine, I don't think that helps too much.
Only when the AIs can do the work of entire companies, end to end, do things really start to change. Because then companies go under. Creative destruction is going to be the primary source of innovation, not adoption. AI is too alien to the white collar world, outside software. Only then do Gary Marcus and Ed Zitron have their faces melt (intellectually). So next year probably, as you say.
I think this is wrong. At the end of the day, successful companies are likely among the most effective ways to organize humans beyond Dunbar's number, it is just that employee alignment is a hard problem.
If software companies as large as Google which are 10x as productive as Google could exist, they would eat Google's lunch. But humans are not ants, and most of the productivity in a company will not be spent on increasing shareholder value but on furthering the individual interests of agents.
There is this neat little thing called capitalism, where a lot of companies are competing for efficiency. Innovations which improve the efficiency of production chains happen all the time without the entire production chain having to change. Stuff like "ok, we will still use drills to make holes in rock to dynamite it, only we will use steam drills instead of manual drills." If a role can be reasonably replaced by AI (and even if it can not), some company is going to replace that role by AI. If this increases their fitness, that will eventually become the industry norm.
Google is only as big and as weird as they are because they have The Money Fountain of search, which they enjoy because they really were very efficient over 20 years ago and their rivals were busy shooting themselves in the foot. But now it's their turn to be the lumbering incompetent titan.
Google can smash a company 10x more efficient by deploying 100x more resources and exploiting their gigantic reach. Or buying the startup that might threaten them, bribing other companies to use Google search by default. If Google had to compete like a normal company, they'd be laughed out of the room.
Take the field of AI. Deepseek is a tiny company with maybe 1% of the compute resources Google has. They're dealing with sanctions too. But per artificial analysis, Deepseek's best AI is slightly better and much cheaper than Google's best AI: https://artificialanalysis.ai/agents/coding-agents
Google search is an AI product, AI is their core business. Google have spent hundreds of billions on AI research, they invented the transformer architecture but have little to show for it.
Well I was pushing for this at work and got us maybe from 0% to 3% of the potential value of the technology.
They have one thing: A search engine that they've intentionally enshittified so much that quite a few people will just ask eg. ChatGPT because Google results these days are so unbelievably bad that it's less effort to verify potential AI hallucinations than try to find the actually useful search result in the googlestack.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I have only ever managed to work for companies that were run with making money as the primary goal, and lifestyle and socialization as secondary goals. And it's not like I was deliberately seeking out companies that were primarily run to make money. Those are just the companies I happened to get jobs at.
Anecdotal evidence, but it makes me kind of wonder about the accuracy of your statement.
Likewise, I've somehow managed to never get one of those so-called "bullshit jobs" that are supposedly everywhere.
We all walk in different circles (even different countries), so my anecdotes are no more valuable than your anecdotes, true.
More options
Context Copy link
More options
Context Copy link
I think changes come primarily with new startups who genuinely need to be efficient.
Indeed, there is a rise in one-person companies in sectors where AI can do work.
Given the timing I wonder how much of this spike due to PPP fraud.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Eventually, the EA people’s personal lives will be in the tabloids, and it will make the whole ‘AI safety’ thing look ridiculous to most of the public.
Those stories are already out there, from the weird sex cults (yes, that was part of the coverage of the FTX scandal) to the weird spiritual cults (this story from 2026, this one from 2023).
Honestly, "Sam Altman is just your usual megalomaniac CEO who wants all the money and control" is the most reassuringly normal figure in the current AI landscape in the US, at least if presenting this to ordinary people.
More options
Context Copy link
Eventually? The mainstream already views EA through SBF and Caroline Ellison, and rationalists through Zizians. To the stupid left, Eliezer Yudkowsky is a sex-obsessed weirdo (apparently acceptable to mock), Scott Siskind is an evil race scientist, and Scott Aaronson is an evil Zionist. To the stupid right they're probably all radical lefties.
The mainstream doesn't really have any idea who these people are whatsoever.
The average normie has maybe heard of SBF before tangentially, but it's a vague memory and they didn't really pay attention to it, it was just another story of Rich Guy Did Some Financial Bad that they might have seen a headline over. And maybe they've heard of Siskind but it's also mostly on vague and hazy terms.
They don't know who Yud is, they don't know who Ziz is, they don't know who Aaronson is.
To most people who have heard of them, SBF and Ellison are crypto scammers, not EA people.
More options
Context Copy link
Yudkowsky is definitely going to be better known than Scott. The latter has heavily shied away from any kind of podcast stuff while Yud has gone all in with the launch of his book.
Which isn't to say that normies know anything about him, just that he's probably the best known of your list
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Such as? The public stuff is already pretty eyebrow raising.
More options
Context Copy link
Closed, ahem, bedroom doors?
More options
Context Copy link
I don't know. "High-end escorts fuck him for free," doesn't exactly make Nate look bad.
Nah, they’d have a field day with Aella.
Aella has already received several softball interviews from decently big outlets. It'd need a big change for them to turn on her.
If AI safety becomes a partisan issue(as it probably will), then conservative media will start releasing Aella lore. Probably NYpost, then OneAmerica news takes it and runs with a 'TDS AI safetyists are controlled by a pedophile-sympathizing hooker' angle.
More options
Context Copy link
More options
Context Copy link
And get branded SWERFs for that.
I think a full publication of Aella's beliefs and activities would shock normie libs it's just the outlets have no reason to do a hit piece on her.
More options
Context Copy link
More options
Context Copy link
Won't be the first
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I feel like a capability by goodness compass is a better lens than the traditional political compass. Capability: is AI rapidly accelerating towards superintelligence or is it a valueless stochastic parrot? Goodness: is AI good or bad for society?
When you do that, the "anti-AI Left" shows a deep fissure, even though both parts are far on the AI-bad side. You have the stochastic parrot wing (led intellectually by Gebru and Bender types) who think it's a water guzzling racist scam run by a bunch of TESCREAL tech bros, and then you have the Eliezer types who think it's going to paperclip the world. They abhor each other and have distinct class, sex, racial, and, uh, neurotypicality profiles. Ant and OAI, and their associated external safety organizations, fall more on the middle of the AI good/bad axis and high but not maximally so on the capability axis, forming a continuum with the hard Yud wing.
Impressionistically, the US doesn't have a major group on the AI-good, maximal capabilities area (some of the Chinese labs, on the other hand, seem to occupy that spot). You have VCs and non-lab tech CEOs who are moderate on capabilities and high on AI-good.
And US politicians? They aren't really thinking deeply about AI. Obama released a statement either today or yesterday, and true to form it was entirely vacuous. Trump is on the AI-good side, but almost entirely to the extent it gives him the economic breathing room for his policies. And capability-wise, all he knows is that AI is never going to be as high IQ as him.
Nvidia is probably the AI good capabilitymaxxing US company, no? Although they're in a weird position now where Huang is both "AI super intelligence will take over all economic activity (except Nvidia chips)" and "AI is just a normal technology that won't replace everyone's jobs" in order to play their different stakeholders.
Similarly, they're also in the position of telling Trump how important it is for the US to beat China, but also they really need to sell all their chips to China.
My general feeling is that Nvidia is probably less motivated by ideology than most of the other actors. Most AI labs are culturally somewhat downstream of LW and have some ideological leanings. Nvidia is just an ancient niche chip company, akin to a random land owner on whose land half of the world's gold supplies were just discovered.
You have a very strange idea of "ancient" if you think a company founded 33 years ago is "ancient".
For a chip maker, they are not ancient, sure.
For a tech company, it is a respectable age, older than most in FAANG.
For a company that central to the AI boom, it is ancient. Anthropic is five, OpenAI ten, DeepMind 15.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
If we're using a 1-10 capability scale, Nvidia is probably a 7. Clearly believes AI is capable and improving, but more focused on chip grubbing than the existential issues people at a 9 (e.g. OAI, Ant) foresee rapidly being thrust on us.
Thinking about it, Larry Page might be a genuine 10-10 AI good, AI capable, though his AI good is something alien to most people (see the accusation he threw at Musk of being speciesist when Musk objected to human replacement). He's fortunately not engaged much operationally with Google nowadays (though still holding substantial governance power).
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The discourse didn't make sense to me until I realized that the pro-acceleration+anti-Anthropic faction consists entirely of venture capitalists who want unregulated open weight models so they can use them to found B2B SaaS startups. Just a complete failure to grasp the implications of the situation we are in. If they fully understood what it means for Anthropic (or OpenAI) to solve alignment and build superintelligence then they would be the ones begging the government for a slowdown.
This would be weird, because open weight models which can do most of what the big three can do exist. I expect that SaaS is not exactly like finding exploits in existing codebases, where the proprietary models made good use of their few months of advance over the open weight models. If my startup model is to write smut on demand for my customers specific combinations of kinks and characters (which is probably not B2B, unless their kinks are really weird), then it does not really matter too much if I run on Mythos or Kimi K3. I would expect that most niche use cases are similar.
The big AI labs probably spend a lot of tokens identifying domains where their model's capability advantage is really decisive (like Navier-Stokes, or exploit-finding) and then a lot more tokens solving problems in these fields to showcase their model.
They're paranoid about the big labs convincing the government to ban open weights (caveat: just because you're paranoid doesn't mean they aren't after you).
More options
Context Copy link
More options
Context Copy link
I am anti-Anthropic because I think Anthropic is clearly run by and staffed by people who are deranged, and thus couldn't produce an aligned AI under any circumstances (same for open AI of course). Thus anything that leave a possibility of them being supplanted is good, and regulatory capture would stop that possibility, whence it is bad.
More options
Context Copy link
"It turns out that this whole faction is actually just motivated by [monolithic sardine-can-compressed explanation of self-interest]" doesn't work for frontier lab Kremlinology and it doesn't work for the /accs either.
In case anyone thinks I'm making this up:
New York Post - OpenAI and Anthropic oversold AI security breaches to pressure feds into protecting turf: insiders
Reading the headline, you might assume that the "insiders" referred to here are anonymous employees of OpenAI and Anthropic. This would be incorrect. All of the "insider" sources are executives of SaaS AI wrapper companies.
The outgroup homogeneity claim is still beneath addressing in detail, but I laughed irl at Kumar clearly using AI to write his quote:
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm more charitable. I think they have started to understand that by default, Anthropic will extinguish all their B2B SaaS nonsense like a hurricane extinguishes a candle flame. In this situation there's just nothing to do but give all your money to Anthropic and maybe the upstream supply chain, until Anthropic buys it out. They'd rather maintain some optionality.
David Sacks is now endorsing voluntary slowdown at the top 2 labs.
This seems awfully bullish on Anthropic considering that OpenAI is at their same level, and other labs are only maybe 6-12 months behind.
This is fair, the logic of capability applies to all frontier labs (or more precisely, companies with frontier LLM research and gigawatts of long-term contracted or owned compute, ie OpenAI, Anthropic, Google, Meta, xAI). They all will have RSI sooner or later, the first two and maybe three have it already.
The difference is mostly in Amodei's political posturing plus Anthropic's obvious preference to not let others use their best models unencumbered. OpenAI does not have a model stronger than Astra that's ready for general use (they have that monster that proved Navier-Stokes, but it is math-specialized, and internally they still use Astra); the gap between the end of development and general availability is measured in weeks, at most a couple months. Anthropic considers users to be a bootstrapping phase.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link