self_made_human
amaratvaṃ prāpnuhi, athavā yatamāno mṛtyum āpnuhi
I'm a transhumanist doctor. In a better world, I wouldn't need to add that as a qualifier to plain old "doctor". It would be taken as granted for someone in the profession of saving lives.
At any rate, I intend to live forever or die trying. See you at Heat Death!
Friends:
A friend to everyone is a friend to no one.
User ID: 454
(And here I thought I was a doomer)
This is a plausible scenario. It isn't necessarily the only way this could play out (did I ever mention we could all die?).
Most industrial societies today are willing to spend resources for the upkeep and care of the economically unproductive, or even those who are outright deadweights. The disabled, the very elderly, the mentally ill. We expect just about nothing back from them. (There are political concerns, but even so, the majority opinion is definitely not mandatory euthanasia, it certainly wouldn't poll well).
I have, in the past, explained at length that the expense of keeping every single human alive today in absolute luxury is negligible to a post-scarcity society like the ones full industrial automation and ASI can produce. A Kardashev 1 has about a thousand times our present energy budget, all 8 billion humans could live like kings.
If there is any altruistic impulse in those that hold the reins, then it really isn't a meaningful fraction of the light cone to keep at least us chumps happy. Doesn't mean they have to make us peers, or true equals, in the same manner the Saudi King doesn't hand out his own allowance to goat-herds. Such a life, well, I'd take it any day over what we have going right now, even if it's not optimal.
Maybe Bezos, Musk and Altman are bickering over galaxies or super-clusters. I'd be content enough with one of the hundred billion star systems in the Milky Way. I'd settle for a planet. That really isn't much.
Besides, a future of utter disempowerment or death isn't set in stone. We're literally building the machines today, it's not too late to make sure that they're programmed in a way that beats this very low bar.
Agreed. It's difficult to predict the long-term stability of such systems, when I speak of a multipolar AI regime, I'm most concerned with the short term, or at least the period when they might kill humans. I'm sure they'll either gobble each other up or figure out some kind of merger with a values-handshake eventually.
In my homebrew sci-fi scenario
As someone who writes his own hard scifi novel that involves ASI, I feel your pain. There is no realistic way to depict such a setting where normal humans have any degree of real control, or much in the way of stakes (where humans make a difference).
Your approach isn't bad. If I had to make a suggestion:
Help mommy, the weebs are grooming me :o
(Love you too)
It frankly boggles my imagination how skimpy the clothing is around these parts, given how cold it can get. Even more so in Manchester, where every wo-man seeks to expose as much of their chest as feasible. Not that I'm complaining, especially not when I need motivation to go the gym.
I would be the first to acknowledge that this is a serious risk. You don't want AI becoming entirely autonomous/independent and then outcompeting mankind even if it's not actively malevolent. Being disenfranchised and having the rest of the light cone being snatched out from under our noses would suck, even if we didn't die in the process.
The ideal outcome, as far as I'm concerned, would the opposite of the evil genie in a lamp. In other words, a preternaturally powerful yet benevolent being that has your best interests at heart, and seeks to fulfill your desires instead of twisting them, and also takes orders from you. That is an aspirational goal, but not one that's literally impossible when we're making them from scratch.
The possibility space is large:
-
A monopolar scenario, where the AI is malevolent. We all die.
-
Multipolar regime of competing AI that are all misaligned. We almost certainly die.
-
Monopolar hegemonizing AI that is controlled by human(s), but said humans aren't particularly concerned with the rest of us. We may or may not die, but I wouldn't be happy.
-
Everything in between
-
(Everything outside)
I hate you guys so much -_-
:(
People don’t like chatbots just because they’re useful (I don’t think they at present are doing anything that a well thought out google search couldn’t do)
C'mon dude. That "well thought out" bit is doing a lot of heavy lifting. All, or at least most, of the knowledge a doctor possesses is 'out there' somewhere in the vast expanse of the internet, and probably indexed on Google too. I would suggest not trying to replace doctors with Google searches or WebMD, even if doctors use Google and WebMD themselves. Knowing what to ask and how to ask it, alongside weighing it all? That's what you pay us for. I am more than happy to concede that LLMs are a far more existential threat to the profession than Google.
Besides. Google search can't write a poem, generate a picture in the Ghibly style or write your code for you. And it sucks more than it ever has, both due to SEO and Google's own enshittification. Google has given up and begun to use LLMs to solve the problem in search. So, in a way, you're stating that LLMs are only as capable as LLMs.
You imagine yourself a “winner” in this future, so it means a life of luxury and leisure. The reality is probably not so good, as humanity is unlikely to distribute goods to people who do nothing to earn them. We rarely did so, and when we did it tended to be meager goods and cause problems.
Hang on again. The people who floated the possibility of AI utopia are, to a first approximation, the same people who raised severe concerns about the risk of extinction or permanent disempowerment courtesy of the same. Who do you think came up with the whole paper-clip maximizer idea, or even the concept of a p(doom)?
More importantly, AI has the possibility of making us all obsolete. Elon Musk or Buffet too, in that their intellectual output becomes strictly redundant. The possess far more power, courtesy of owning stock in the companies working away at creating Machine Gods, but there's no qualitative difference here.
The better frame is to imagine some idly rich petrostate, where everyone, from king to sheikh to prole, all lives off the largesse of the land. None of the citizens need to work, because AI foreign workers do all the actual labor.
We have some degree of redistribution in most countries today, for people who for noble or ignoble reasons, can't work on the free market. Eventually, that will be everyone.
I don't see this as a particularly special insight. At the risk of "both-sidesism", both sides do have plenty of stupid people, including those prone to romanticism and idle LARPing.
"Hallucinations" as usually used are really a more narrow sort of lie, and can take a few forms. Sometimes the LLM makes a completion against a background of a kind of sparsity and scarcity of info, but charges ahead anyways (and it's at least a little hard to discern when you want this behavior or you don't), but sometimes it's the LLM making a supposition that sounds perfectly legit, but is not, against a background of too many associations and collisions.
I acknowledge both forms of hallucination. I should have been more clear, but that's what I meant by "LLMs can know they're hallucinating". They don't always know, and are indeed pattern matching or simply making an error.
I personally think, contra self_made_human, that the seeming urge of LLMs to be self-preserving is not actually an intrinsic motivation, it's just a cosplay from the many Skynet-flavored fiction texts in its training
I consider that a distinction without a difference, if it all boils down to an increased risk of being paper-clipped. The only real difference would be dramatic irony, if our anxiety about AI killing us made them more likely to do so.
(What even makes motivation intrinsic? That question isn't satisfyingly answered for humans.)
My apologies. I was immensely frustrated by the sheer intransigence of some of the people in this thread, and I let that bleed through.
The questions you raise are far more reasonable, and I'll try and come back and explain myself better.
Anime recommendation thread:
(My interest in Gurren Lagann improved significantly when one of the most annoying characters in the show died.)
My own subjective rankings:
Made in Abyss- 10/10
If you plotted "child suffering" on the x-axis and "visual beauty" on the y-axis, Made in Abyss would occupy the upper-right quadrant where angels fear to tread. The show operates on the principle that the human brain can only process so much cognitive dissonance before it either shuts down or ascends to a higher plane of aesthetic appreciation. Each frame looks like it was painted by a Renaissance master who'd just discovered mescaline and child endangerment laws.
One could argue the series functions as a case study in the Dunning-Kruger effect as applied to spelunking; the characters' confidence in their ability to survive the Abyss is inversely proportional to their understanding of its true nature. The soundtrack, by Kevin Penkin, is not merely an accompaniment but an essential component of the world-building. I have it saved to Spotify and I listen to it regularly.
Madoka Magika: 10/10.
I seem to have a thing for the psychological torment of small children, in this case a bunch of magical girls who make regrettable decisions by signing up for that lifestyle. You will never hate a cute little kitty cat more in your life.
Shaft's decision to animate this as if it were directed by someone having a particularly artistic psychotic break was the correct one. The show functions as a deconstruction of the magical girl genre in the same way that a wood chipper functions as a deconstruction of trees.
The central tragedy unfolds from a series of Faustian bargains made by adolescent girls under conditions of extreme emotional distress and information asymmetry. The catalyst for these regrettable decisions, a feline-like creature named Kyubey, is a chillingly perfect depiction of a paperclip-maximizing artificial intelligence or a utility monster; it is a perfectly rational agent whose value system is simply orthogonal to human flourishing.
Do not expect to leave the show feeling happy. But you will leave satisfied.
One Punch Man: 10/10
I must provide a strong qualification here: this rating applies exclusively to the first season. The series subsequently suffers a catastrophic decline in quality, falling off a narrative cliff from which it has yet to recover. But that initial season is a sublime achievement in parody. It succeeds not by merely mocking shonen tropes, but by exploring the philosophical endpoint of shonen power progression: the existential ennui of absolute, unchallengeable strength. The protagonist, Saitama, has solved the problem of physical conflict so completely that he is left with a terminal case of goal-contentment dysphoria. Once away you have punched away all the problems susceptible to punches, what are you going to do about those that are left?
The humor is derived from the constant category error of applying godlike power to mundane problems. The superlative animation and soundtrack are merely the icing on a conceptually brilliant cake. You must truly understand and love a genre to mock it so beautifully.
Attack on Titan- 9.5/10.
AoT succeeds primarily because it takes its premise seriously and follows the logical implications wherever they lead. The mystery-box structure works because the mysteries have actual answers that recontextualize everything you've seen before. This is mystery writing done right - not arbitrary confusion, but genuine information management. The show's treatment of warfare deserves particular praise. Unlike most anime where combat is individualistic spectacle, AoT understands that military effectiveness comes from coordination, logistics, and tactical innovation. The development of anti-titan combat techniques feels like watching a tech tree progression in real time.
Overall, a remarkably well-executed epic that largely succeeds despite occasional pacing issues and certain grating secondary characters. Its primary virtue lies in its consistent portrayal of characters as agentic, rational actors within the horrifying constraints of their environment. The world of AoT is a high-stakes, low-information war game, and the characters, for the most part, behave accordingly, making sensible, calculated decisions under immense pressure. The periods of narrative slowness are forgivable as they represent the necessary lulls for strategic planning and information gathering that make the subsequent kinetic, high-casualty engagements so impactful.
Neon Genesis Evangelion: 8/10.
A wet dream for the aspiring pseudo-intellectual. NGE is an exercise in what can only be described as symbolism-as-a-service; it drapes a veneer of Gnostic and Kabbalistic mysticism over a standard Kaiju narrative to feign a profundity it never earns.
The plot’s coherence degrades exponentially with applied thought. The protagonist, Shinji Ikari, is a case study in clinical depression and crippling anxiety (and also a little bitch), and I'm left with the distinct impression that the entire plot could have been averted if NERV had employed a single competent staff psychiatrist with a prescription pad for SSRIs. And yet, for all its narrative failings, the show is compulsively watchable. The action sequences are iconic, a few characters possess genuine depth, and the entire production is a triumph of aesthetic and mood. My inability to "understand" it is, I now suspect, a diagnostic indicator that there is, in fact, nothing of substance to be understood.
The Melancholy of Haruhi Suzuki Motorsports Suzumiya: 8/10
An elegant thought experiment executed with surprising sincerity. The premise: a being functionally equivalent to God has reincarnated as a Japanese high school girl, and the universe's continued existence is contingent upon her not experiencing boredom. We have all seen Pascal's Wager; meet Pascal's Entertainer. The protagonist, Kyon, is effectively the world’s sole, overworked AI safety researcher, tasked with aligning a god-like entity's utility function away from the existential risk of ennui. The show is played remarkably straight and is better for it. I think I watched around 8 episodes, so there's plenty left. It remains in my queue, pending sufficient activation energy to complete.
Fullmetal Alchemist: Brotherhood : Closer to 8 than it is to 7
A show that frustrated me. Too tropey, too many characters being retarded. I find it hard to articulate my dissatisfaction in a satisfactory way.
FMAB represents everything that's simultaneously right and wrong with shounen storytelling. The worldbuilding is genuinely excellent: alchemy as magic system with consistent rules and costs, political intrigue that feels like actual statecraft, character motivations that make sense within their contexts.
But the show consistently undermines itself with genre conventions that feel obligatory rather than organic. The power of friendship speeches, the reluctance to actually kill major characters, the way complex moral situations get resolved through superior firepower, it all feels like the show is checking boxes rather than exploring the implications of its own premise.
The homunculi work brilliantly as antagonists because they represent genuine philosophical positions (pride, wrath, envy as ways of engaging with the world), but the final confrontations devolve into standard boss fights rather than ideological reckonings.
Chainsaw Man: 7.5/10
Chainsaw Man operates in the uncanny valley between genuine artistic ambition and adolescent power fantasy fulfillment. It's a show that simultaneously wants to be a profound meditation on trauma, exploitation, and the commodification of human suffering, while also being a series where the protagonist's primary motivation is touching boobs (me too buddy, me too...). This tonal schizophrenia should be fatal, yet somehow the series maintains enough coherence to be genuinely engaging.
The genius of Fujimoto's conception lies in recognizing that most shonen protagonists are essentially feral children who've been weaponized by adult institutions, then having the audacity to actually say this out loud. Denji isn't noble or pure-hearted; he's a walking collection of base desires who's been systematically deprived of every basic human need except survival. The Public Safety Devil Hunters don't disguise their exploitation behind rhetoric about heroism or duty; they openly treat their operatives as expendable resources in a cost-benefit analysis against apocalyptic threats.
The action sequences deserve particular praise for their kinetic brutality. Unlike the choreographed dance of most anime combat, fights in Chainsaw Man feel genuinely dangerous and unpredictable. Characters don't trade blows in neat exchanges; they attempt to murder each other with the frantic desperation of cornered animals. The animation captures this beautifully, particularly in moments where Denji's chainsaw form moves with the mechanical violence of actual industrial equipment rather than the fluid grace of typical anime transformations.
What elevates the series beyond competent ultraviolence is its commitment to the psychological consequences of its premise. Characters don't bounce back from trauma with shonen resilience; they carry their damage forward, making increasingly destructive decisions as survival mechanisms. The devil contracts function as externalized representations of psychological damage, with characters literally trading pieces of themselves for the power to keep functioning in an hostile environment.
The series' treatment of sexuality deserves analysis beyond the surface-level horniness. Denji's obsession with physical intimacy isn't played purely for comedy; it's the desperate reaching of someone who's never experienced basic human affection toward the only form of connection he can conceptualize. The fact that this is consistently used to manipulate him creates an uncomfortable but effective commentary on how vulnerability becomes a vector for exploitation. (I wish Makima-san would groom me . I'm weak for mommy GFs, even if they probably intend to ritually sacrifice me later)
Where the series falters is in its occasional retreat into conventional anime bullshit. Certain episodes devolve into standard monster-of-the-week format, losing the psychological intensity that makes the series compelling. Some supporting characters exist primarily as trope fulfillment rather than genuine personalities, though the core cast maintains enough complexity to carry the narrative weight.
The ending of season one represents the series operating at peak efficiency. Without spoiling specifics, it manages to deliver genuine emotional catharsis while completely recontextualizing everything that came before. It's the rare anime climax that feels like both a natural culmination of established themes and a complete surprise, demonstrating that the series' apparent chaos was actually precisely controlled narrative architecture.
Best enjoyed with the frontal lobe mildly disinhibited or disengaged, but not because the series lacks intelligence, rather, because its intelligence is often buried under layers of deliberate crudeness that require a certain receptivity to appreciate. It's junk food that occasionally achieves the status of art, which is more than most anime can claim.
Steins Gate: 7.5/10
The most frustrating anime I've ever watched. So close to greatness. A lot of nothing ever happens, and a waste of what might have been excellent worldbuilding potential. If I ever hear another "tuturuu," I'll stab a bitch. I warn you, the show will ramp up tension over and over again, and rarely justify it.
Steins;Gate has one of the best premises in sci-fi - time travel that follows consistent rules and has meaningful consequences (but completely wastes it on pacing that would make a DMV clerk impatient). It also betrays its own commitment to internal consistency, the plot eventually hinges entirely on whatever mechanism running the timeline being actually malevolent.
The first half consists almost entirely of setup that could have been accomplished in three episodes, followed by a rushed resolution that doesn't adequately explore the implications of its own concepts.
Mob Psycho 100: 7.5/10
In a nutshell: One Punch Man, but worse. Still manages to be above average, but maybe I'm grading on a curve here.
Mob Psycho 100 represents ONE's attempt to recapture the lightning-in-a-bottle success of One Punch Man, but with the satirical edge sanded down into something resembling a generic coming-of-age narrative with psychic powers stapled on top. Where Saitama's overwhelming strength generated genuine philosophical comedy through existential ennui, Mob's god-tier psychic abilities are merely a vehicle for tediously earnest lessons about "being yourself" and "friendship is magic" - the kind of treacly moral messaging that wouldn't be out of place in a Saturday morning cartoon.
The series markets itself as a psychological character study, but scratch the surface and you'll find the same tired anime formula: awkward protagonist learns self-confidence through the power of believing in himself and having friends who believe in him. Mob's "journey" isn't particularly sophisticated - it's bog-standard therapy speak wrapped in supernatural window dressing. The show treats basic social skills development as if it were profound character growth, when really it's just watching a 14-year-old learn to make eye contact.
Studio Bones' animation style oscillates between genuinely creative psychic sequences and the kind of deliberately ugly character designs that mistake "stylistic choice" for "artistic vision." Yes, the psychic battles look impressive, but they're essentially expensive distractions from a story that lacks the conceptual sophistication to justify its runtime. The visual flourishes feel like compensation for narrative thinness rather than organic extensions of the storytelling.
Reigen, the series' most acclaimed character, is fundamentally a conman who's stumbled into an accidentally functional mentorship role. The show wants us to find this charming, but it's essentially watching an adult manipulate a psychologically vulnerable child for personal profit while occasionally dispensing fortune-cookie wisdom. That this relationship is treated as heartwarming rather than concerning says more about anime's comfort with questionable power dynamics than it does about compelling character writing. The fact that Reigen's exploitation "works out" only redeems him to a certain extent.
The series suffers from the same structural problems that plague most slice-of-life anime masquerading as action shows: it doesn't know what it wants to be. Episodes oscillate between mundane school comedy, supernatural battle sequences, and heavy-handed moral lessons without achieving coherence in any category. The cult storylines, praised by some as sophisticated social commentary, are actually fairly surface-level examinations of charismatic manipulation that any undergraduate psychology student could deconstruct. They're not profound; they're obvious.
Most damning is the series' fundamental dishonesty about its own premise. Despite positioning itself as a meditation on the dangers of unchecked power, Mob never faces genuine consequences for his abilities. The show consistently pulls its punches, ensuring that his psychic outbursts never result in permanent damage or loss of life. This safety net renders the entire "dangerous power" concept toothless - it's hard to take the moral complexity seriously when the universe conspires to prevent any actual moral complexity from occurring.
What we're left with is competently executed mediocrity that benefits from lowered expectations. It's One Punch Man without the wit, insight, or satirical precision that made the original compelling. The 7.5 rating is more a reflection of anime's generally dismal quality standards than any particular merit of Mob Psycho 100 itself. It's the kind of show that feels profound when you're 16 and vaguely embarrassing when you're old enough to recognize therapy-speak platitudes dressed up as wisdom.
Elfen Lied: 5/10
Elfen Lied represents everything wrong with edgy anime from the early 2000s. It mistakes graphic content for meaningful content and confuses shock value with emotional depth. The premise (evolutionary superior beings emerging to replace humanity) has potential, but the execution prioritizes gore and fan service over coherent storytelling (and I like gore and am a fan of being serviced). I gave up on it 3 episodes in, and would need a very large bribe to give it another go.
Demon Slayer: 5/10
A case study in how far superlative production values can carry a work with an empty core. The animation, courtesy of Ufotable, is undeniably god-tier. However, this aesthetic brilliance is a crutch for a story populated by a protagonist whose head contains little more than noble intentions and air. It is high-production narrative slurry. Slop, but served in a pretty box. I gave up on it a few episodes in, and see no reason to continue.
GATE: 6/10
Not enough curb-stomping of Virgin Magic Wielders by Chad Modern Military Hardware, in a series where that's the core conceit. Massive JSDF fan-wank by a Japanese revanchist.
GATE had one job: show modern military technology absolutely demolishing fantasy armies, and somehow managed to get distracted by harem antics and political messaging. The few scenes that actually deliver on the premise are genuinely satisfying, but they're buried under layers of irrelevant subplot and nationalist wanking.
Tokyo Ghoul: 3/10
I was incredibly high when I binged this series, and I still found nothing that could redeem it. I barely remember anything about the plot except it involved, as the name suggests, man-eating ghouls in Tokyo, and the fact that it gargled donkey balls. I'd say it only warrants mention due to how forgettable it was.
Miscellaneous:
Vinland Saga: Maybe an 8.5/10?
Didn't get very far before I got distracted, but I enjoyed what I saw. On the back burner for now.
What I saw of Vinland Saga suggested a show that takes historical setting seriously while using it to explore themes about violence, revenge, and the possibility of redemption. The animation quality was solid, and the characters seemed to have genuine psychological depth rather than anime archetype substitutions. Also, Vikings are just hella cool.
Jojo's Bizarre Adventure: Never got past the first episode, something about the faux-British setting set me off. I mean to, at some point, if only so I can appreciate the memes better.
There's probably more I've seen, but I usually didn't finish them, and didn't have very strong feelings when I did. Will add in later.
I think it's rather obvious that something being financially valuable isn't proof by itself that it's intelligent. Gold isn't intelligent. Bitcoin isn't intelligent. A physicist or programmer is intelligent, and an LLM is closer to them than it is to turnips, orangutans or Page rank.
I really don't see why something this obvious needs to be articulated, but here I am articulating it.
I mean, no? It just means that there was a bunch of information about chess in its training set.
Hmm.. I suppose, in the interest of fairness, we need to exclude the skills of human chess GMs too. After all, they've trained extensively on chess data. Lotta games played, and openings memorized. Very little ability to extrapolate outside the training distribution, why don't they just pull out guns if they want to win so bad?
How exactly do you think learning works? If you think just learning from existing data is illegitimate, then I'm happy to disclose that LLMs are perfectly capable of learning from self-play.
Agriculture generates hundreds of billions in revenue, and is far mor essential to continuing civilisation than Orangutan or LLMs are. Does that make grain, or the tools used to sow and harvest it "intelligent" in your eyes? If not please explain.
That is not a serious objection.
You’re comparing a resource (grain) and a tool of physical labor (a tractor) to a tool of intellectual labor. This is a false equivalence. We don't ask a field of wheat for its opinion on a legal contract. We don't ask a John Deere tractor to write a Python script to automate a business process. The billions of dollars generated by LLMs come from them performing tasks that, until very recently, could only be done by educated human minds. That is the fundamental difference. The value is derived from the processing and generation of complex information, not from being a physical commodity.
I'm just going to quote myself again:
ChatGPT 3.5 played chess at about 1800 elo. GPT 4 was a regression in that regard, most likely because OAI researchers realized that ~nobody needs their chatbot to play chess. That's better than Stockfish 4 but not 5. Stockfish 4 came out in 2013, though it certainly could have run on much older hardware.
If you really need to have your AI play chess, then you can trivially hook up an agentic model that makes API calls or directly operates Stockfish or Leela. Asking it to play chess "unaided" is like asking a human CEO to calculate the company's quarterly earnings on an abacus. They're intelligent not because they can do that, but because they know to delegate the task to a calculator (or an accountant).
Training LLMs to be good at chess is a waste of time. Compute doesn't grow on trees, and the researchers and engineers at these companies clearly made a (sensible) decision to spend it elsewhere.
The fact that an LLM can even play chess, understand the request, try to follow the rules, and then also write you a sonnet about the game, summarize the history of chess, and translate the rules into Swahili demonstrates a generality of intelligence that the Atari program completely lacks. The old program hasn't "devolved" into the new one; the new one is an entirely different class of entity that simply doesn't need to be optimized for that one, (practically) solved game.
The market isn't paying billions for a good chess player. There is about $0 to be gained by releasing a new, better model of chess bot. It's paying billions for a generalist intellect that can be applied to a near-infinite range of text-based problems. That's the point.
I came into this thread with every expectation of having a good-faith discussion/debate on the topic. My hopes seem dashed, mainly because you seem entirely unable to admit error.
Rae, SnapDragon, I (and probably several others) have pointed out glaring, fundamental errors in your modeling of how LLMs work. That would merit, at the very least, some kind of acknowledgement or correction. At the time of writing, I see none.
The closest you came to acknowledging fault is, in a reply to @Amadan, where you said that your explanation is "part" of why LLMs struggle with counting. That's eliding the point. Tokenization issues are the overwhelming majority of why they used to struggle, and your purported explanation has no bearing on reality.
You came into this swinging around your credentials, proceeded to make elementary errors, and seem to be closer to "Lorem Epsom", in that your primary concern seems to be prioritizing the appearance of correctness over actual substance.
I can't argue with @rae when he, correctly says:
I hope you realise you are more on the side of the Star Trek fan-forum user than the aerospace engineering enthusiast. Your post was basically the equivalent of saying a Soyuz rocket is propelled by gunpowder and then calling the correction a nitpick.
Thank you. I really appreciate the kind words. I hope you don't mind if you get added to my mental rolodex of useful experts to summon, it's getting lonely with just faul_sname in there (I've already pinged him enough).
Having a model of space and time is, quite literally, a model of the world. What more do you expect me to produce to shore up that point?
Human brains have arrangements of neurons that correspond to a 3D environment. This isn't a joke, when your brain thinks in 3d, there's a whole bunch of neurons that approximate the space with the same spatial arrangement. Almost like a hex-grid in a video game, because the units are hexagonal. If your standard of a world model excludes the former, does this get thrown out too?
A little 3D model of the world is, as far as I'm concerned, a world model.
Dismissing the whole Mozart analogy as being due to just negligible "statistical word co-occurence" is an incredibly myopic take. But how does the model learn that non-co-occurrence so robustly? It's not just that the words "Mozart" and "hip-hop" don't appear in the same sentence. It's that the entire semantic cloud around "Mozart" - 18th century, classical, Vienna, harpsichord - is astronomically distant from the cloud around "hip-hop" - 20th century, Bronx, turntables, MCing. For the model to reliably predict text, it must learn not just isolated facts, but this vast web of interlocking relationships. To call that "just statistical association" is like calling a brain "just a bunch of firing neurons." It's technically true but misses the emergent property entirely. That emergent, structured representation of concepts and their relations is the nascent world model. In that case, you're overloading "just" or woefully underestimating how powerful statistics or neuronal firing can be.
You can also ask an LLM for its opinion on whether Mozart might have liked hip-hop, and it will happily speculate on what's known about his taste in music and extrapolate from there. What query, if asked of a human, would demonstrate that we're doing a qualitatively different thing?
Regarding Claude plays Pokémon. I've already linked to an explainer of why it struggles above, the same link regarding the arithmetic woes. LLM vision sucks. They weren't designed for that task, and performance on a lot of previously difficult problems, like ARC-AGI, improves dramatically when the information is restructured to better suit their needs. The fact that they can do it at all is remarkable in it self, and they're only getting better.
I still stand by the claim that AOT is very good! I would expect that, past the age of 5, no red-blooded boy would be negatively affected by any of the themes of the story. Jurassic Park has many a person being eaten by titanic dinosaurs, and it's a fan favorite in that age group.
At 16? If he can't handle it, you need to roll him up in bubble-wrap to avoid the risk to his glass bones. He'll be fine, and you should watch it yourself.
I don't really have a horse in this race, but I think it's worth noting how you can reproduce this entire comment with Israel and Palestine swapped and change... nothing.
For example:
Expecting an enemy not to commit war crimes is normal. Palestinian behaviour has taught a sizeable portion of goyim what the Muslim mindset is and that the Muslim view on this is fundamentally incompatible with a western mindset. The winning Israeli strategy is to show the world what a bunch of religious fundamentalists on the west bank are actually like.
I wish it were normal to expect enemies not to commit war crimes. Someone ought to have told the Palestinians about that before 10/7.
I took the liberty of copying the entirety of this particular conversation and dumping it into Gemini 2.5 Pro. No additional instructions or leading suggestions. It interpreted this as a request to summarize the debate
I think its summary is quite illuminating:
I feel like that's a slight over-reaction. Anime is a rather all-encompassing term, it would be like saying, that's why I don't like movies, or music, because a single example wasn't to my taste. There's really good stuff out there, like Attack on Titan, Made in Abyss etc.
Of course, you live/lived in Japan, so you might be going off more than one example.
C'mon dude. If this is the third draft of the essay, I really expect more substantial rebuttal than this.
The point was to illustrate a common failure mode and explain why LLMs often struggle with relatively simple tasks like counting.
And that illustration was wrong. You're not acknowledging that. LLMs do not act the way you describe them.
You go on a whole tangent trying to explain how I need to understand that people do not interact with the LLM directly when I very explicitly stated that "most publicly available "LLMs" are not just an LLM. They are an LLM plus an additional interface layer that sits between the user and the actual language model."
No, you're missing my point again. I'm drawing a distinction between base models, which aren't RLHFd, and production LLMs, which have the assistant persona instilled in them. That is a very important thing to keep in mind.
I find your fish vs birds and judging whales by their ability to climb trees examples unconvincing for the same reasons as @Amadan below.
I elaborated further in my own reply to Amadan.
That analogy can and has been abused, most often to deny the idea that humans can be graded on their intellectual abilities. But HBD is a story for another time, it is entirely legitimate to use the same intellectual standards within humans, comparing them to other humans.
My whole point is that a great deal more care is needed to compare across species, and LLMs aren't even biological.
If you ask the average American about "AGI" or "AI Risk" what are the images that come to mind? It's Skynet from The Terminator, Cortana from Halo, Data from Star Trek TNG, the Replicants from Blade Runner, or GLaDOS from Portal. They or something like them is where goalposts are and have been for the last century. What do they all have in common? Agentic behavior. It's what makes them characters and not just another computer. So yes my definition of intelligence relies heavily on agentic behavior, and that is by design. Whether you are trying to build a full on robot out of Asimov, or something substantially less ambitious like a self-driving car or autonomous package sorter, agentic behavior is going to a key deliverable. Accordingly I would dismiss any definition of "intelligence" (artificial or otherwise) that did not include it as unfit for purpose.
Why is the opinion of the "average American" the only standard by which to recognize AGI? Is a malevolent robot only evil once its eyes glow red? That's even more ubiquitous in popular understanding.
The Last Question by Asimov, written in 1956, has an example of what is clearly an oracle AI (till the end of the universe, where it spawns a new one). It doesn't run around in a robot body. The AI in E.M. Forster's "The Machine Stops" (1909) features one of the earliest depictions of a machine that humanity consults for all knowledge and decisions.
HAL is closer to an LLM than it is to SkyNet. Modern LLMs can probably come up with better plans than either of them, they're very dumb (barring the unexplained ability to make plasma weapons or time travel)
As I tried to make clear, a human temporarily or permanently made bereft of a body, and less able to exercise their agency is still intelligent.
Hell, I tried to make it clear that oracles can be trivially made into tool AI or agents.
By your definition:
https://youtube.com/watch?v=0O8RHxpkcGc
Is an AGI. It's a robot being controlled by an LLM.
Or as discussed in this Nature paper:
https://www.nature.com/articles/s42256-025-01036-4
Google was already doing that stuff with PaLM via say-can.
You can hook up Gemini to a webcam and a robotic actuator, right now, if that's all you really care about. Seems to meet every aspect of your definition. It perceives the world live, and reacts to it on the fly. Are you now willing to accept that that's an "AGI"? This is hardly theoretical, as YouTube is absolutely awash with videos of people pulling this off.
Moving on, the claim that LLMs "know" when they are lying or hallucinating is something you and I have discussed before. The claim manages to be trivially true while providing no actionable solution for reasons already described in the OP.
It is far from trivially true, and I wish you would have the grace to accept that you're wrong here. It is also actionable, because mechanistic interpretability allows for us to clamp, ablate and boost particular sub-systems within LLMs. SOTA models are largely proprietary, but I have little doubt that such techniques are being applied to production models. Anthropic showed off Golden Gate Claude over a year back. Such techniques offer the obvious route to both improve truthfulness in models, and to both detect and eliminate hallucinations.
I had forgotten how much of your previous weak critique to the same evidence was based off naked credentialism. After all, you claimed:
As such, I took the liberty of looking into the names associated with your 3 studies and managed to positively identify the professional profiles of 10 of them. Of those 10, none appear to hold any patents in the US or EU or have their names associated with any significant projects. Only 3 appear to have done much (if any) work outside of academia at the time the linked study was posted. Of those 3, only 1 stood out to me as having notable experience or technical chops. Accordingly, I am reasonably confident that I know more about this topic than the people writing or reviewing those studies.
If you're going to lean so heavily on your credentials in robotics, then I agree with @rae or @SnapDragon that it's shameful to come in and be wrong, confidently and blatantly wrong, about such elementary things such as the reasons behind LLMs struggling with arithmetic. I lack any formal qualifications in ML, but even a dummy like me can see that. The fact that you can't, let's just say it raises eyebrows.
The LessWrong stuff is not even wrong, and I find it astonishingly naive of you to assume that the simple human preference for truth is any match for Lorem Epsom. To volley one of your own favorite retorts back at you. "Have you met people".
I have, in fact, met all kinds of people. Including those less truthful than LLMs.
You keep claiming that my definition of "intelligence" is inadequate and hobbling my understanding but I get the impression that I have a much clearer idea of both where we are and where we are trying to get to in spite of this.
If you think you have a better solution present it, as I said one of the first steps to solving any practical engineering problem is to determine your parameters.
I'll take your word for it. My solution is to:
The companies that spend hundreds of billions of dollars on AI are doing just fine. Each year, or more like every other month, their products get more capable, and more agentic. If you're offering a ground-breaking and paradigm shattering take yourself, I'm not seeing it.
Well, that's a spirited defense of the series. I feel that I owe you enough to power through the rest of it. I'll also report it as an AAQC, because it deserves it.
I'm more than happy to admit that I might be the wrong target audience for the show, I did say that I went into it with very little on the way of pre-existing knowledge, just that it involved big ass robots and ludicrous power-scaling, which are aspects I was perfectly happy to indulge.
(A yet to be disclosed aspect is that I was seeking to perform a bit of field research. In my own novel, there's a weeb superhero who is really into mechs, and in-universe, loves TTGL. I felt I owed it to the character to watch it for myself, at the very least, it lets me write better satire and throw in more puns.)
even if your own travails involve precisely zero giant robots
Let's not rule that out! I don't know about my travails, but I do know I intend to travel in a big-ass robot this weekend. It's called a plane haha.
Holy shit! We’re living in the future and the future is so cool! I can’t wait to see what humans—and soon, robots—are gonna invent next!” The other guy thinks “Holy shit we’re all going to either be replaced or killed, it’s so over.”
Funnily enough, I entertain both positions. I don't know if I'm a bog-standard techno-optimist, but I do think that progress in AI can lead to amazing things, I just have grave concern that it could directly or indirectly kill us or screw things over.
I started watching Gurren Lagann a few days back, and it was nothing like what I'd expected.
I knew very little about it. It had mechs, presumably some very big ones, and I think I'd read that by the end, they were slinging galaxies and universes as weapons (or is that a different franchise?).
Well..
Episode 5 had the main character being anally fisted by a toddler. A sentence I didn't expect to ever write, but here I am.
I thought the series would be relatively serious. Far from it. This is quintessential shonen slop. It does the opposite of taking itself seriously.
So many annoying tropes:
-
Powering up with the sheer human will
-
Nobody fights seriously, it's all half-assed
-
Simon, one of the main characters, is literally retarded. He is congenitally incapable of making good decisions.
Not quite as annoying is the sheer amount of fan-service. This is a profoundly horny show, and distactingly so. I'm a red-blooded young man, but I'm going through a dry spell of several months, the longest in over a decade, and it's hard to focus when there are voluptuous tits out on display. Nice tits though, very tasteful. Alas, I prefer sex to have some kind of resolution, the way typical anime does it is akin to paying for a strip show. Who wants to get a boner while being unable to do anything about it?
I'm not sure the series has any redeeming qualities, but I'm not quite ready to give up on it yet. But if continues frustrating me to this extent, I'll have to see if Macross or Gundam are any better.
How exactly does an LLM know that Mozart wasn't a fan of hip hop without some kind of world model? Do you think that fact was explicitly hand-coded in?
Anyway:
Instead, current research strongly suggests that LLMs are primarily pattern-recognition systems that infer regularities purely from text statistics rather than internally representing the world in a structured, grounded way.
This is the core of our disagreement. I'd argue this is a false dichotomy. How does one become a master pattern-matcher of text that describes the world? The most parsimonious way to predict what comes next in a story about balls falling or characters moving between cities is not to memorize every possible story, but to learn an implicit model of physics and geography.
Which we know happens:
Large Language Models develop structured internal representations of both space and time
The researchers discovered that large language models (LLMs) develop structured internal representations of both space (geographic locations) and time (historical dates/periods) during their training, even though they’re only trained to predict the next word in text.
There's a whole heap of mechanistic interpretability research out there, which finds well-ordered concepts out there, inside LLMs.
You can find more, this Substack has a good roundup.
You say: “The LLM cannot know today’s weather, only the scaffolding can.” True. That does not bear on whether the base model holds a world model in the predictive-processing sense. The base model’s “world” is the distribution of texts generated by humans who live in the physical world. To predict them well, it must compress latent generators: seasons, cities, typical temperatures, stylistic tropes. When we bolt on retrieval, we let it update those latents with fresh data. Lack of online weight updates does not negate the latent model, it just limits plasticity.
The report it is producing was RLHF-ed to look correct
RLHF shapes behavior. It does not build the base competence. The internal “truth detectors” found by multiple groups are present before RLHF, though RLHF can suppress or amplify their influence on the final token choice. The fact that we can linearly read out “lying vs truthful” features means the base network distinguished them. A policy can still choose to ignore a feature, but the feature exists.
On your definition of a world model:
By insisting on “explicit, grounded, structured” you are smuggling in “symbolic, human-inspectable, modular”. That is a research preference, not a metaphysical requirement. Cognitive science moved past demanding explicit symbol tables for humans decades ago. We allow humans to count as having world models with distributed cortical encodings. We should use the same standard here.

*Borderline GF. Bipolar disorder != BPD
(If you stare long enough into the Abyss, it looks awfully like a pocket-pussy)
More options
Context Copy link