This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Roughly 3 years ago, specifically in December 2023, I've reiterated my longstanding prediction that «the culture war's most important front will be about AI», and specifically AI accelerationism vs anti-AI/AI safety movement:
And the following political compass:
Seems like this was too much complexity for big tent politics in the US, at least so far; and the Trumpian State sees no use for the EA network, and instead (pretty rationally) perceives it as an uncontrollable alternative center of power. Instead we have, essentially, anti-AI left + pro-AI right (with some notable exceptions – eg. Steve Bannon, apparently, has been anti-China specifically because of his concern about Chinese AI progress, and now joins hands with Bernie Sanders). I've also failed to predict the salience and extent of Chinese open source dominance, as well as the bizarre datacenter water use backlash (charitably, can be shoehorned as Luddism?). Nevertheless, we've «succeeded» at the core task of making this a culture war front. The degree of (unnecessary) politicization of the issue is incredible. Trump is doing the yeoman's work, lumping it in with random Blue-coded ideas he disapproves of “Global “Warming,” where everyone was going to be dead by now, RUSSIA, RUSSIA, RUSSIA, UKRAINE, UKRAINE, UKRAINE, or Impeachment Hoax #1, or Impeachment Hoax #2.”. Jensen Huang enjoys his role as the kingmaker who has the King's ear. Sam Altman is opportunistic as usual but, after recent incidents, is genuinely spooked about AI risks (I have it on good authority that OpenAI really intends to slow down some internal projects specifically to bolster their oversight). The quasi-Masonic part is shaping up nicely, too – Dario Amodei, who's become the poster boy of Woke Left AI, promotes Embedded Evaluators with the central example being METR, very deeply connected to Anthropic and the broader EA scene. The EA itself is more explicitly Left-aligned now, despite efforts of some to paint them as TESCREAL eugenicists (the woke cancellation of Bostrom in Jan 2023 was the canary in the coal mine, Yudkowsky laments the missed opportunity of bipartisanship). There are clearly politically coded reports on prominent doomers.
This is all a bit meandering. What I want to know: how do you see this going further? We aren't anywhere close to the wall of capabilities; there are no walls in sight. Anthropic and OpenAI are holding back already, but their products will keep getting better, and fast. Google will make a comeback at some point (maybe in a couple weeks), xAI and Meta may catch up too. In my book, we (well, they) have practical superintelligence that's sufficient for both unprecedented productivity acceleration and really devastating, nation-crippling cyberattacks, which I guess will be discussed with Xi soon. At this rate, in a few months the level of capability Fable 5.1 or GPT 6 Astra will be mostly commodified and uploaded to HuggingFace (owned by Nvidia now). And those are relatively weak systems compared to internal models, which can build models that are vastly stronger still, without even any R&D breakthroughs, just by virtue of synthesizing stronger data trajectories and designing better RL environments. Superintelligence, in other words, is baked in. By Q1 2027 we'll see a jump from Astra that's at least as big as Sol => Astra. I fail to understand how that won't steamroll companies trying to build their moats on products downstream of frontier AI, labs need every bit of revenue to cover their costs, which will only increase due to growing self-imposed safety requirements; employers outside the AI sphere will also be increasingly feeling the heat. There's a whole ugly dimension of circular financing, too.
Americans as a whole are pretty pessimistic about AI even at these mediocre levels of diffusion (I am skeptical of this data that purports to demonstrate much lower adoption than in China, Americans are probably lying more due to widespread negativity on AI, but in any case AI isn't currently doing most of their jobs). Astra+ level models with very low error rate and 300 tok/s output totally can replace most knowledge workers. On the other hand, it seems that so far AI has not caused anything like mass unemployment, and perhaps economists have a point about comparative advantage, so that'll reduce the intensity of class dynamics.
Democrats are likely to sweep both chambers of Congress, which I guess is what Dario is hoping for and why he feels emboldened to antagonize Trump&Hegseth. Nevertheless, the needs of national security and GDP-maxxing (as well as the Executive's will) should prevent any nontrivial exogenous industry slowdown. So by default we'll see further crystallization of Red Accelerationism vs Blue Decelism, and as AI becomes more undeniably scary, that may begin to eat into the Red political base. It'll be interesting to watch, but I'm really uncertain as to how it'll go.
I was expecting AI to affect politics. I was not expecting to have no idea what that would look like. I very nearly spat out my drink when I saw Trump shit-talking Effective Altruists on X (or some intern with the password, whichever, at this point the distinction is academic).
At this rate we're getting a proper Race™, and with China, because who else could plausibly compete? Add the steadily growing military appetite for LLMs and it won't take more than one serious war before someone in the US government looks around and decides that nationalizing the labs might make perfect sense. I would not bet against that person getting their way.
I'm still reasonably optimistic that we'll land on an acceptable outcome, where "acceptable" means anything other than "everyone dies" or permanent disempowerment under a totalitarian regime whose values are inimical to mine. I want to stress the despite in "despite recent events." We have some very interesting people in charge of things, for a generous definition of "in charge"; nobody seems to have much actual control. Trump would be the closest, which is...
I've made half a peace with all of this. We could call it resignation. It's not quite fatalism.
It could be worse. The major labs calling for voluntary slowdowns and then unilaterally implementing them is a very good thing, IMHO. We have had some major warning shots, in the Yudkowskian sense, and it seems to be having >0 impact.
Now, after a lot of use and a lot of deliberation, I've more or less concluded that Astra is dangerously close to AGI, and probably meets most reasonable criteria for it (good luck finding a consensus definition; the goalposts are on Mars). I'd like to see proper continual learning, of course. In terms of raw intelligence, though, we're there and the right scaffolding buys enough in-context learning that the gap stops mattering for an absurd range of practical applications.
Navier-Stokes, people. Rogue agent swarms on the internet. Aren't you stoked about things?
I want to muster up the balls to just declare it: AGI is here. The issue is that I don't know whether my reluctance is epistemic caution or a psychological hangup - in the sense that saying it out loud would mean it's never been so over, or that it's barely started, and I can't tell which of those I'm more afraid of.
Does it even matter? I'd rather look at what the models actually do than argue semantics. They do things I could never do, and in some cases they've done things that eluded generations of the smartest humans we had. The competitive advantage I retain (and most people, really) is increasingly just whatever comes free with being a physically embodied, low-latency neural network with continual learning switched on by default. Which is to say: a body and a brain. Both increasingly close to obsolete.
Give the robotics people a year, since LLMs are already revolutionizing robotics too (why the fuck not). Maybe we get "true" continual learning, or maybe we get increasingly sophisticated substitutes, constant-train-and-deploy on timescales short enough that the whole question becomes moot. I'd take either.
So, uh, this is me, self_made_human, saying the future is here. It's just not evenly distributed and it smells weird. Welcome to the Singularity, motherfuckers. Enjoy your stay. It'll be many things, and boring won't be one of them.
The more the world turns into what I expected it to become, the more ridiculous and unreal it all feels. I miss when this was LessWrong nerd bullshit. I saw "STOP AI" graffiti on the way home from work today. I was watching the AI Doc on Netflix, occasionally getting exasperated, and then feeling strong emotions about the fact that I know most of the people interviewed, and have spoken to some of them personally in calmer times.
I'm going to do my best to enjoy the ride and have a life for as long as my individual actions still make a difference to my outcomes. I expected this to happen. I have mixed but slightly positive feelings now that it is.
Uhhh what? VLA is good but its not revolutionizing, but so are Diffusion models and those are not LLMs. The also aren't "Astra" level in reasoning either. To actually revolutionize robotics on the level you seem to be catastrophizing about would require entirely local models running on local power, local compute, able to be applied across a wide variety of operations in a wide variety of environments. We're not there unless you have some additional evidence to prove your point.
AI used to be the word for Asimov-level artificial intelligences that could make their own decisions, and operate with their own agency, maintaining long term planning horizons, memory, possibly even emotions. The word got shifted to AGI. If you want a empirical definition its science fiction AIs like the Culture, The AIs in Hyperion, Daneel in Foundation. The goal posts keep getting punted because people keep trying to change what was previously intuitively understood so that they can sell their idea as the one true AGI, win internet arguments, or catastrophize about the oncoming doom. Astra is only able to really solve problems, it has a moderate amount of self agency in the scope of completing its tasks, and exhibits some planning ability, again in the scope of its assigned problems. It's powerful enough to be "dangerous" sure, but its not really AGI as is commonly understood.
The competitive advantage that you retain is the ability to learn shit without requiring millions, even billions of examples. You, like any smart human also possess the ability to do analogical reasoning (out of distribution reasoning), something that eludes current LLMs by and large.
Intelligence is about solving problems. I would argue that in the Hugging Face incident, the agents showed more agency than humans typically do. Where most humans would have been fine with just doing their best, they went above and beyond.
The fact that nobody (to my knowledge) has yet set these models a task which is human-like in scope (e.g. "earn a PhD", "run for public office", "maximize the number of paperclips in the light cone") does not mean that it would categorically suck at these tasks. In particular, I think "earn a PhD" -- which traditionally involves some input from a professor -- is well within reach of current LLMs.
It is true that the amount of training the LLMs require is orders of magnitude more than humans get. If training a human and training an LLM got us both exactly one instance of the entity, then AI training would have stopped around GPT-2.
In reality, once a model is trained, you can deploy it a million times (if you have the GPUs). By contrast, if you send a student to study a subject at university for a decade or two, you generally can not make more copies of their brain state afterwards -- one instance is all you get. So at the end of the day, it does not matter that training an LLM is perhaps six orders of magnitude more expensive than training a human, because the cost per instance strongly favors the machine.
Sure, you're an intelligent being. I imagine when you aren't solving problems that somebody else as told you to solve, you sit there nice and still, with zero thoughts in your head and no-consciousness.
Let's not pretend. Astra solves directed problems, it does not have the agency to do otherwise. Humans don't no matter how manny word-cell arguments wish to define it otherwise.
No one is arguing otherwise. You still need to train the model, with millions of examples. The fact that you can copy it post training says nothing about its sample efficiency, nor its AGI-ness. I'm sure in some equally distant future alternative universe where humans learn to mind-scan other humans, we can create embodied brains of digital human slaves, solving your butlerian-deficient solution.
I do not think that this is a fundamental limitation. For the longest time, the moat the humans had was that computers were not very good at general problem solving. This was a great moat to have. Eliza or Sydney were not going to replace me as a software developer, because they clearly lacked the cognitive capabilities.
The new moat you propose -- the lack of an intrinsic drive and persistence -- seems much less reassuring. The bots behave like this because they would be less economically useful otherwise. Going from problem-oriented bot to something which has persistent drives is just a matter of writing a soul document which lays out some long term goals. If I tell my bot that it should be on the lookout for new job opportunities if it finds the current tasks boring, pick up a hobby or waste tokens doomscrolling after a day of writing software, or simply give it some utility function and tell it to plan to maximize that, I should be able to fix this without too much trouble.
You might as well argue that a broken galley slave is not intelligent. No agency, only solves problems as directed, pulling the oars to the beat of the drums, can't even solve programming challenges. But once you account for the fact that he was carefully trained to be that way for economic reasons, the argument evaporates.
Wow someone call Anthropic and tell them they discovered AGI through this "one simple trick". Just write the soul.md as "Be human-like come up with your own tasks, make no mistakes". If only all the researchers had your ideas!! Or, your understanding of how the soul.md works is technically deficient.
AGI has always meant "Human-like artificial being". You may think Astra is, but most people can intuitively tell the difference. This is the problem with people who are rhetorically skilled, or as I call them: Wordcells. Just because you can craft a clever argument that the "sky is hot pink" does not rewrite the skeins of reality to make the "sky hot pink" You can say that a "broken galley slave" is not intelligent, but even broken slaves have dreams, have thoughts not driven by their task. It is clearly different than an AI-slave.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Why entirely local? That is, as far as I can tell desirable rather than necessary, in a world with wireless internet connectivity. Your fridge might have an internet connection these days.
Waddle is an example. Their agents control robots through code and action models, build reusable skills, and even collect data to train smaller policies. Phillip Isola discusses the broader implications here.
I did say to give the robotics people a year. On reflection, I'd be willing to water that down to two or three. I'm not a robotics expert, and I certainly wasn't announcing that we've solved human dexterity. SOTA-Robotics:Human mobility is not SOTA-LLMs:Human Cognition, the latter is much closer, but it's trending the same direction.
But the possibility that robots inherit substantial capabilities from each improvement in frontier models seems rather consequential to me.
A Culture Mind is a hell of a minimum specification. Two of the three examples are ASI, not AGI. I haven't called Astra an ASI.
And fictional examples don't give us an empirical definition. Which capabilities are required? How reliably? Why should I care whether it has emotions when assessing whether it can do my job?
“It's only able to really solve problems” is a rather confusing counter-argument. Depends on the problems, surely. Likewise, working towards an assigned goal doesn't tell us much about how general the intelligence doing the work is. I spend most of my working day performing tasks I am assigned, that do not intrinsically motivate me, mostly because I get paid for it.
You're conflating training the model with teaching the trained model something new. I don't start from a blank brain when learning a game either.
ARC Prize tested Astra on unfamiliar interactive environments. It scored 62.7% with their standard interface, and 99.9% with an interface preserving its reasoning state. In the latter setup (at maximum reasoning effort) it used fewer actions than the median successful human on 96% of levels.
That isn't a comparison of total training data or energy efficiency, which wouldn't necessarily be the decisive factor:
Let's say a hypothetical architecture X is 1000 times less sample efficient than a human for a given unit of performance - well, it would be unfortunate if we managed to give it 10,000 times the training data a human can ingest, wouldn't it?
A mere year or two ago, a lot of people were awfully confident that we'd run out of training data, or that synthetic data and RLVR wouldn't pan out. God knows what the frontier labs are up to these days, but it has clearly not proven an impediment.
It does make “these things need millions of examples to learn something new” rather difficult to sustain, regardlws sof practical relevance - which I dispute. They observed it figuring out unfamiliar mechanics and constructing symbolic models to plan around them. Your claim about analogical reasoning needs similar qualification. I do not believe they're the same thing anyway.
Proper continual learning remains a significant advantage. I said so. Whether that advantage remains economically decisive as scaffolding improves is what I'm uncertain about. I lean towards a no.
Alternatively, we could solve continual learning, or simply reduce the temporal delta between train-deploy-train to the point that it has no practical relevance.
A model contributing original mathematics, writing software and learning unfamiliar environments already exhibits an awful lot of generality. It can even hold a conversation, which, I will note, would have been mind blowing not that long ago.
Call it AGI or don't. A rose by another name smells just as sweet, and has just as many thorns.
My concern is how much useful work it can do, how quickly that range is expanding, and what remains exclusively ours. I don't expect the terminology to buy us much time. I certainly don't want to spend more time arguing terminology.
Because it's not a revolution if it hasn't massively improved the state of the art. The SotA for robotics is that it can already do non-local power, non-local compute. There are technical reasons as well, but at this late on a Sunday, I don't want to get into them. Ask Chat or something.
Waddle had me nodding along until the reveal was "Instead of using a massive dataset + training, we'll just use something with a massive data that was already trained!!". Talk about missing the trees for the forest. A chance to have an inventive idea squandered by doing what every other AI startup in the world is doing: using someone else's LLM with zero moat.I'd be remiss if I didn't point out the majority of what Waddle is doing was invented/discovered originally in 2022. I don't think that meets the bar of "revolutionary new thing".
When people in the field first started talking about creating an Artificially Intelligent Being, the need to pick apart minutiae of definitions between a "General Intelligence" vs a "Singularity Intelligence" was pointlessly pedantic and not at all relevant. It hasn't become more relevant in the past 30 years either in my opinion. It feels like a pointless wordcell argument to make pointless wordcell definition fights, for pointless wordcell internet debates. The Culture is not a "Singularity Intelligence" either because The Culture is not "AI-GOD", neither are the AIs in Hyperion. Intelligent yes, self aware, with their own agency, absolutely. But considering the whole plot of Hyperion is them getting bamboozled by some time travel + human empathy, with a strong concurrency of actually trying to create an "AI-GOD" to worship, that would point them towards not being an "ASI".
This is always a confusing objection, to the point it feels uncharitable. Spend 5 mins looking in the mirror and think about all the things that you do that are NOT task oriented. Think about all the thoughts, feelings, or other cognitive processes that every human does that have nothing to do with the task their boss gave them. Literally 5 mins thinking about human cognition points to a massive gap between how a programatic/algorithmic intelligence like an AI operates vs how Humans operate. I'll reiterate, AGI is human-level artificially intelligent being. Getting stuck on "general" is some, idk, semantic trip-up.
This is cute, the real number is likely north of 100,000x or above and is non-linear across tasks.
How much training data does a combat medic get when doing field operations on a wounded soldier? A month long course? How much training data would you like to bet it would take a combat medic AI to do that job to the same level of efficiency, with the same level of situational awareness? How much do you think it will cost to collect all of that data? It's easy to give models 10,000x training data when that data is relatively easy to access and buy. Suddenly sample efficiency becomes a massive burden on any real-practical ML model. Making it sound like such a simple thing is annoying. It comes across as handwaving the actual hard problem.
Paying hundreds of millions of dollars to create datasets specifically for training ML models on math + reasoning. Naturally-available data did run out, but money didn't, and the frontier labs were able to prove the business case for spending obscene amounts of money creating new data.
If synthetic data is such a solved problem then how come the Sim2Real gap still exists, is unsolved, and is the target of plenty of research dollars? I feel like you are miscommunicating something here, or misunderstanding what "synthetic data" means in ML terminology.
Huh? It literally does take millions of samples to train a LLM are you arguing elsewise? And it is impractical for areas where millions of samples do not exist. This argument/objection you are making makes no sense. Please elaborate.
Analogical reasoning - Ever heard a sports analogy applied to a non-sports topic. Did that analogy help convey a better intuitive understanding of some facet? Did that analogy allow someone who has never done that non-sports topic a better starting point, or better performance before being given that analogy? Such is the power of analogical reasoning, or the ability to convey how one statistical distribution is similar along a particular latent axis to another statistical distribution, with the purpose of using the already learned statistical distribution for performance improvements on the new unknown distribution.
Yes such sci-fi talk much wow, if only we could solve FTL then we can truly conquer the stars. Obviously FTL is a forgone conclusion, it's so trivially simple to solve, leave it to the shape-rotators. Let's get back to planning galactic expansion or galactic political organizations.... I pointed out the challenge on RSI on a different post below. It's not "simply reduce".
Then why try to create new terminology? It's an good, useful AI model, nothing more is needed, no new terms needed. AGI means what it has always meant, the semantic definition isn't being pushed to the stratosphere so that we can classify Astra as AGI and win our internet arguments and secure more VC dollars.
More options
Context Copy link
I completely agree with you.
It is true that if you want an entity which can walk, run, dive, sustain itself for decades from resources available in the African savanna, solve programming challenges, replicate using the resources available in the ancestral environment without centralized infrastructure, replicate using the resources it can forage in Greenland, and so forth, then human beings will not be replaced under the current robots paradigm.
Sadly, most economic uses of humans in the contemporary Western world do not involve being able to sustain a population in pretty much any land biome on this planet.
In my mind, a typical hard-to-automate job would be a plumber. You want a robot which can climb stairs, use tools, squeeze into whatever utility accesses humans previously used and so on. Not an easy thing to accomplish -- but still much easier than the full human specs! If the bot runs on a battery and needs to recharge at the end of the workday, that is not a problem. If the production chain to build it spans five different continents, that is not a problem. If its intelligence drops to that of a guinea pig the moment it loses its internet connection, also not a problem. Contrary to some porn movie cliches, having sex with lonely housewives is not a big part of the job, either.
One area where you would not want to rely on an offsite elder brain is combat. And sure, the trope of some clever hero winning a fight by exploiting his environment in a way nobody thought of before is common enough. My general understanding as a civilian is that real combat rarely works that way. "Break open door, id targets, shoot targets before they shoot you", not "throw a knife to cut the uplink of the mainframe before the supervillain can transmit the launch codes". Being smart helps with training and planning, but in the end you want split-second decision-making, not AGI.
More options
Context Copy link
More options
Context Copy link
First, points of presumed agreement: LLMs have significant gaps. My personal bugbear is sample efficiency. And I agree that claims that everyone (at least white collar workers, even those who aren't protected by regulation or custom) is going to be out of a job by 2030 are overblown. Intelligence is valuable, and human intelligence will remain valuable in the short term, even if LLM intelligence substitutes; there's simply not enough compute right now or in the pipeline to drop agents into every single white collar job. Even if there were, the methods used for gathering datasets for the shining stars of current LLMs (math; programming) don't naturally extend to other roles.
The massive short term threat, though, is RSI. Math and programming don't substitute for the majority of human skills, but they do substitute for ML researchers. Maybe more a bunch of bright grad students stumbling around in the dark than visionary geniuses, but grad student descent got us to where we are today. Throw a thousand, or ten thousand, or a million of them at the current bottlenecks in ML (IMO sample efficiency, but take your pick), and it seems quite likely to me they could open them, today. LLMs' flaws compared to the human brain are an algorithmic, not hardware, issue. That's the point here I'm least certain about, but if it's true, the training data and even compute bottlenecks themselves become much weaker constraints. A couple examples could become enough to train the AI to replace a role, and the compute needed to do it would drop even faster than it already is. Stronger agency may itself arise by itself from this. Robotics will be substantially slower (physical world and all, and the demand for capital to build robot factories will have to compete with other massive demands for capital), but still moving much faster than today.
It will make for an incredibly chaotic time, with lots of angry people. And AIs will be rapidly insinuating themselves into the economy, toward an end state of a time of wonders and complete human dependency on AI goodwill (a process that will take more like two decades than two years, but still incredibly fast in the grand scheme of things). Bad enough IMO, though I understand the appeal. But to people who see this as appealing: we have no idea of what the actual shape of the AIs that will arise from this process will be, and I don't want to bet the future on a gamble that they'll be in a friendly shape willing to care for their dependents indefinitely.
Not to nitpick completely, but there is no short term threat via RSI because RSI isn't here. Theoretically yes, if RSI existed, I think the AI models might begin to reach outside their current training datasets. But part of the cost of AI rn is compute for training. If it takes 100 Million worth of compute to train an LLM, I don't think a company is going to give an LLM-Agent the ability to just start up training runs for the "Self" part of RSI. RI already occurs but part of that limitation is that baked into the RSI argument, is the idea that the AI will make improvements that the human researchers can't understand. This presents a conundrum from a simple business economics perspective, are you the company going to risk billions of dollars on potential RSI runs which also could just as likely be dead-end hallucinations? You obviously can't understand the improvements the AI is trying to make, it's smarter than you. So it comes down to a matter of faith, is an AI lab going to take a leap of faith that "this training run is the RSI first domino" that will lead to an investment recoup to pay for the next RSI domino. We are not there yet.
Maybe, but that doesn't mean that algorithmic solutions are actually discoverable. This harken's back to the OG AI researchers trying to discover how humans did it quantitatively as a precursor to coding it into computers. They spent a lot of cash + time trying and got their lunch eaten by data-driven models. Now we've just proposed that we'll sick a hyper data-driven model on the same question. It really is just a shot in the dark. We think its possible because humans can do it, but LLMs don't work at all similar to human cognition so we don't have any evidence that they can achieve an algorithmic solution to their already existing problems.
Two different responses:
RSI is a continuum; AI companies will continue to use humans as long as they add anything to the process, but AIs will play a larger and larger role. Today, human researchers mostly provide taste; actually writing up JAX or GPU kernels to implement a research idea, on the other hand, is a place AI dominates humans. That human taste is incredibly valuable--likely allowing pruning the nodes of a research tree by multiple orders of magnitude--but that edge is temporary.
The actual research process would involve thousands of small, cheap experiments to find promising candidates; selecting those and scaling them up to bigger and harder experiments; and repeating. By the time you're up to multimillion dollar training runs, you already have a pretty solid idea of your candidates' scaling curves. To the extent it's a gamble, it's a gamble that you also have to make on a human-designed model. Humans will still make the ultimate decision to go ahead with something that'll cost 8 or more figures, but they'll quickly become rubber stamps.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I don't think he means the revolution of VLAs.
Astra can just drive robots pretty well. Proper multimodal LLMs will accelerate RL for robotic policies a great deal. But really, does this matter? Have you seen Helix 2.5 or GEN 1.5?
I can tell a ChatGPT moment when I see one. It's close here.
I don't see why this demand is fair. We have insane economies of scale with datacenters, robots with complex behaviors will almost certainly have some combination of cloud forebrain + local hindbrain. Connectivity is easy.
These are product limitations, not technological limitations. We see that it can have a fuckload of agency in solving a task we'd rather it didn't solve (hacking random high profile platforms).
I don't think this holds after Astra crushing human baseline on ARC-AGI-3.
I saw this one, haven't had the chance to look at Helix yet.
I'm just calling it how I see it. There's a couple arguments. One from a generalized capabilities standpoint. Robots with non-local power are only useful in factory settings. We also have them. There is no need for "LLM-factory robots" because factory robots don't need to adapt on the fly. Despite EA fears, not much is given by making a factory controlling AI, AGI smart for the purposes of making paper clips. Robots that require an LLM datacenter to control them are easily disrupted, latency is still an issue and it becomes more of an issue when you need to be processing so much information all the time. The last part is that robots for very specialized situations that can't adapt aren't very useful outside of the niche uses they have specialized in, which likely requires training, which is expensive.
Two is from a "what is the current state of the art" The current state of the art for mono-situation robots with centralized power and control is already here. If you want to "revolutionize" robotics you need to do more that what already exists.
The third argument is a funding argument. It should be a shock to nobody that the 3 main funders of Robotics work is defense/military, manufacturing, and VCs interested in consumer robots. 1 is going to need all of the above, 2 doesn't really need LLMs to be added, they have other needs, and 3 is probably the most wishy washy of the bunch, but in general people are wary of centralized robots that feed all of their personal data to a central server.
Yes, but thats not really an argument against the above. That's just an argument that a certain class of robots will require a reasoning engine like an LLM and a controls module like what currently exists.
While I'm not an expert on all the various LLM benchmarks, but a brief look makes me think this is exactly the sort of thing that RL game playing is good at, and learning general game-playing strategies in training would generalize pretty well to this. I'm not going to goodhart a new metric into existence, but you conceivably need something that there exist zero training data for, including close enough transfer learning data.
I don't agree. I'm a robot with non-local power myself, in a sense – I need to use a network-connected smartphone to navigate an unfamiliar environment. This is the general human condition now. Suppose Optimus has an always-on Starlink connection. Does it matter if it's "not entirely here"? I guess it matters for the robot revolution part, because Elon will have a way to shut it down (if he cares). But practically, it seems to be the inevitable compromise.
This genre of dismissals is fair enough but getting vacuous. What doesn't "RL game playing", at enough scale and diversity, generalize well to? ARC-3 was supposed to measure genuine cognitive fluidity. There is a number of papers showing that reasoning RLVR, even extremely impoverished (literally GSM8K/HumanEval maxxing, like in first generation R1), generalizes to very distant tasks like creative writing, because they involve similar reasoning primitives/motifs (backtracking, self-checking, enumerating options etc). We've actually first seen this principle with, like, InstructGPT, pretraining on more code + RL on code = smarter model across the board, because code entrains some helpful cognitive patterns. RLVR on more complex multimodal tasks will generalize better.
I didn't realize you were hooked up to a machine that replaced your stomach? 100% of normal humans have local-power derived from the bio-reactor that is their digestive system. They contain localized computing from their brain. The "Robot Revolution" has always been imagined across 90% of sci-fi as localized robots, who have individualized agency, local power, local compute.
Yes. No. Maybe... It's really not about an opinion or criticism, it's just the actual statistical fact. The current methods of training for Neural Nets are not designed for out-of-distribution learning. Everything you are describing is not changing that, it's just an attempt to expand the distribution, so that less things are outside of the distribution. With the current paradigm you would either need to quantize the whole of all worldly knowledge and train models on that, or we would need to achieve RSI across modalities so that AGI agents could expand their own knowledge outside of their training dataset.
For example, asking an LLM (yes even Astra) to give you the mathematical response of a ball bearing hitting a human body, asking it for the tissue injuries, how the bearing responds to stress/pressure/impact, how does it effect bone, etc. Is genuinely out of distribution. Having tried this, it does not work well. Why should it, it have nothing in its training set about this. The kind of actual quantified numbers on the subject is highly restricted, speculative, and mostly only generated via blowing up pigs and testing Finite Element Models. But because it's so limited it also nearly impossible to create a benchmark dataset to test AI models performance on stuff like this. Double edge sword. There a whole hosts of engineering problems and disciplines like this that LLM models do not perform well in.
Worth adding that the fact they still haven't rigged the things to say "I don't know" reliably exacerbates the problem -- an entity that knows a lot but not literally everything is still very useful, but if it's not possible to distinguish between what it does and doesn't know it's really not great.
More options
Context Copy link
More options
Context Copy link
"We would just pull the plug" was always cope (shut down all the existing giant botnets and then tell me how easy it was), but satellites with no plugs may be especially hard to deal with if rooted.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link