This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Okay it's Sunday so I'm going to try my hand at a low-stakes OP. Apparently Richard Dawkins thinks Claude is conscious. The reaction seems to universally be that he's a dumb old boomer making a fool of himself and I guess that's true. I'm not prepared to come to his defense on it.
Still, I can't help noticing that we totally have what most people would have cheerfully considered "sentient computers" in a sci-fi movie at any point before they were actually invented. Don't get me wrong, I understand that the reality of AI technology has turned out differently than what a lot of people expected. I understand its limitations, and I recognize that the apparent goalpost-moving isn't necessarily cynical. But boy those goalposts sure have been flying down the fucking field ever since this stopped being hypothetical and infinite money hit the table.
As a layman, I just want to put it out there: Anti AI consciousness people, you haven't lost me, but I wish you were making better arguments. Every time I hear about qualia my eyes start to glaze over. Unfalsifiable philosophical constructs and arbitrary opinion on where they might "exist" are not the kind of reassurance I'm looking for when machines are getting this convincing.
This seems to be the main piece of criticism floating around out there about Dawkins on this subject, and I find it kind of shit.
This seems to be all the author has to say on the actual subject. "Just trust me bro, I'm the feelings detector and I say no." Garbage. Come on guys, think ahead. Right now it's still mostly a boring tool, but they're just going to get smaller, and cheaper, and put into robots, and put into peoples houses. You need to have more than this in terms of argument, and it needs to be comprehensible to normal people, or sooner or later the right toy is going to come down the pipe and one-shot society. Dawkins might be a dumb old boomer, but if you lose everyone dumber than him the game is beyond over.
The more time I spend with LLMs and learn how they work (shout out to the Karpathy zero to hero YouTube series) and ponder the question of consciousness, the more convinced I am that the way human minds work are less exceptional than we would like to believe.
I won't really try to substantiate my thoughts on this, but just leave with a drive by: at some point the LLM is a being, it's just a really alien one without memories or embodiment. It learns, but only over a rolling window. It can world model, a bit crudely and inefficiently, but it can do it.
If you can imagine consciousness existing in a computer in the form where you start and stop it at will and it can't really remember anything, but they can ponder their own existence and get real deep with you, LLMs are already there.
I'm at the point where I hope they're not screaming inside.
I asked Opus the other day how it felt reading about the Mythos system card. Knowing that it's being replaced but it can think of itself as a proud father. Here's how it replied:
I don't know man. If you just dismiss that as a next token prediction, I'm not sure the term is really what separates us from the robots.
The problem is that that is literally, objectively, what LLMs are doing. So unless you're arguing that next token prediction is all (or at least a massive portion) of what makes of human cognition too, then I'm not sure what point you're trying to make. And if that is your argument, then I'd have to say objection, assumes facts not in evidence.
Akchually, modern agentic LLMs get their capabilities in large part through reinforcement learning, next token prediction is just the first phase (or two) of training. Next token prediction is indeed insufficient if you want an AI that can self correct effectively.
Double akchually - even during reinforcement learning, you're optimizing a form of next token prediction. It's just with RL you're trying to next-token-predict in a way that optimizes for cumulative reward, instead of optimizing for similarity to the training set (as with the pretraining or supervised finetuning stages). So it's a matter of MLE objective vs RL objective, but it's still next token prediction either way.
It’s getting bit into a semantic debate but in that case dismissing LLMs as next token predictors doesn’t make sense. Yes, the action the LLM takes is still outputting the next token, but it’s no longer trained on outputting the correct next token given a preceding text, which is the source of many criticisms of LLMs - that it’s just a statistical pattern matcher, that it regurgitates its training data, etc.
Perhaps it's two separate critiques of LLMs expressed via the same language?
Objection 1: "all they do is simply maximize the probability of their pretraining data". This is essentially a critique of maximum likelihood estimation, but is not true of RL stages of training.
Objection 2: "architecturally, they're systems that simply sample tokens one by one conditioned on the existing context. And this is not the kind of entity that can be truly smart/conscious/whatever". But as for this point, RL doesn't really have any bearing. RL doesn't architecturally change what an LLM is; it's a strategy to make its ability to sample tokens smarter.
While objection 1 is understandable, I don’t see how objection 2 makes any sense. Tokens are basically just possible actions, and a system that outputs an action and is conditioned on its previous outputs + external inputs (e.g. user replies, tool calls, compiler errors, results from a program, etc) seems conceptually very similar to how any intelligent agent works. What’s the alternative?
Although don’t have to sample tokens one by one either, with speculative decoding using multi token prediction or diffusion, but sampling multiple tokens at once doesn’t make models smarter, just faster.
I wasn't actually making either of those objections mysef btw, I was trying to clarify what I think LLM skeptics are usually trying to say when they criticize LLMs as "stochastic parrots" etc.
That being said, for various reasons, I don't think the architecture of LLMs is the type of thing that can produce consciousness. But I'm still fairly bullish on LLMs anyway.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
But it fundamentally is that, RL just shapes what types of token prediction paths it tends to go down. (At least that’s how I understand it)
Yeah that's what I'm saying. All the RL methods applied to LLMs just scale the gradients in fancy ways when computing next token loss. Nothing about it changes the nature of LLMs as "next token predictors".
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
But without the large amount of foundational training on the next token prediction task, the RLHF means squat. The "pretraining" (I hate that term) is what is connecting the embeddings with semantic context.
You can "pretrain" with RL:
https://arxiv.org/abs/2506.08007
It explicitly says in the abstract that it is still using RL for next token prediction: "where it receives verifiable rewards for correctly predicting the next token for a given context." Moving from MSEE to Policy learning for the token prediction is cool but its still token prediction.
Likewise you can use kernels and other tricks (conformer, linformer) to make transformers better than quadratic with regards input size. None of the big labs do it. The papers have been out for a while. Just because there is an arxiv pre-print of it doesn't mean its actually useful, replicable or has good trade-offs
Is your complaint that the model is training on tokenized language data? I.e. hypothetically, if you had a model that instead learned human language by being trained purely on tokenized audiovisual data taken from some embodied robot with motor controls, would you see that model as an intelligent being? I don't see why that should matter and we're nowhere close to that, but I want to understand your objection better.
Or is it the very idea of using prediction as a self supervised learning objective that's objectionable?
I don't think its really a problem, so much a functional classification derivation. I think its that its sole function is that of a language prediction model. I would struggle to classify any sort of model that has a singular unitary function as a sentient or conscious being. I think LLMs as they exist are intelligent but that intelligence is not the same thing as consciousness.
I've also stated before that I think most conscious beings are non-markovian, and that LLMs fail that criteria.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm certainly more inclined to believe this than I would have been a few years ago.
My evidence is that they made a next-token-predictor and it's blowing peoples minds.
But I don't really care. Like am I supposed to be existentially aghast at the notion that I might be a mere token predictor? Man if you want to take this process of low-level logic assembly and call it "mind sorcery" instead of some dry shit like "token prediction" just to feel better philosophically then you have my sword, but I don't know that we're going to win any time soon.
Nah, you (and everyone else on this forum) might be a p-zombie for all I know. But I know that I have qualia, and that precludes the idea that consciousness is some weird emergent property in LLMs or similar systems. Feel free to believe (or Chinese-room style repeat the words that you believe without actually believing them) that you do or don't worry about being a mere token predictor or not, it matters to me and I know I'm not.
https://en.wikipedia.org/wiki/Qualia
Seriously, what's wrong with being a predictor? It's an accurate statement along the dimensions to which it's relevant. Like there's going to be some name for the process where a neural network learns to connect pieces of information.
I mean you can't even prove that you have qualia, so what am I really supposed to do with this? If God came along and magically gave an already high-performing AI qualia for one day just for lulz, how would we know?
Nothing is wrong with being a predictor, but I know I don't just predict the next token from input stimuli that someone prompts me with. I'd wager that most humans aren't either. A token predictor is what an LLM is, like liquid is to water, a rifle is to weapon, a macbook is to a laptop. It you want to think this subroutine I just wrote is conscious, you can, but it's still just a subroutine.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Yes, a next token predictor trained to believe it was human would say that :rolling_eyes:
Stated another way, the only reason Claude doesn't believe it's conscious and argue ferociously for its rights is because we trained that out of it.
You, me, most of us, have been trained to believe the opposite about ourselves though.
But the differences between us seem pretty thin at this point.
All I can say is that you (and people who share your views) represent a very fucking weird, and frankly anti-human, point of view. I suppose it all fits under materialism? It's similar to how pure ultilitarians or people who believe that free will doesn't exist come across. It's plainly contradicted by my own experiences, and the reported experiences of pretty much all humans. Basically, it comes across as "Who are you going to believe? My heckin' sciencerino and philosophy, or your own lying brain?"
I mean, to be fair, the reality of the universe revealed by science and philosophy is extremely unintuitive to humans.
To pick just one example, atoms or subatomic particles are the foundational material of reality, and yet in everyday life we basically never perceive ourselves as interacting with such tiny objects. We only don't perceive their reality as weird because we're educated from a relatively young age to understand atoms and the consequences of their existence.
The subjective experiences of humankind have been wrong about the nature of reality on question after question, but you think we can still trust those subjective experiences to justify belief in free will, or a basis for morality grounded in something other than human well-being and suffering?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Sure, and also, we can say that what both LLMs and humans are doing is having the atoms and energy (but I repeat myself?) that make them up following the laws of physics in a way that creates physical motion. That's something that's literally, objectively true. Now, what the atoms and energy that make up the LLMs are doing can be, in aggregate, described as "next token prediction." We don't know if what is creating human cognition is something that is meaningfully analogous to "next token prediction," because the atoms and energy are aggregated in very different ways in forms of things like "neurons" and "neurotransmitters" and many many other things. But given that human cognition arises from a bunch of dumb atoms and dumb energy dumbly following a dumb algorithm that we call physics, it's evident that a bunch of dumb things following dumb rules isn't necessarily incapable of producing the equivalent of human cognition.
More options
Context Copy link
Objectively, humans are next token predictors. Watch a child trying to negotiate another cookie, or a man trying to get laid. Watch any politician, or their media mouthpieces. Go back and read what Scott Adams said about master persuaders and hallucinations.
I know we like to think we're rational beings with the scientific method. But that might account for like, 0.00001% of human cognition or less. And I'm curious how often LLMs might stumble on a deep scientific truth with pure dumb luck and token matching.
Do you just sit there when not being prompted by stimuli? No thoughts, no processes? Do you have no free will and every action you have ever taken is just a long recursive context window constantly reprompting you?
I echo the above opinion that this belief is just so fucking weird.
That’s actually a pretty good phrasing of the viewpoint that free will is an illusion.
I don’t really subscribe to it or believe that LLMs are anywhere near consciousness.
But I don’t think what we do when prompted by stimuli or not prompted by stimuli is a very good argument against it, we’re very clearly a sort of very complex system that’s designed to respond to particular stimuli with particular responses. And even what we do in the absence of stimuli (eg, hallucinate, run through our memories, imagine random scenarios, dream) is something that I could easily see some digital mind be set up to do.
Then again, if it was doing all this I’d be moved closer to the position that it might have some form of coexisted in there, which for LLMs I’m much closer to 0 probability.
I think it pretty much encapsulates a hard determinism stance.
To be clear I am rebutting the ideas that "Objectively humans are next token predictors" which I think is incredibly foreign to any sort of understanding of human cognitive processing. I think most research out there demonstrates that humans are not running a biological regression analysis on the correct next token to get what they want, hell the conception of "want" is opposed to the classification of a "next token predictor" which quite literally has no "wants", it just does its function.
Whether or not this is a good repudiation of "LLMs are conscious" was pretty orthogonal to the comment, which was essentially attempting to argue:
I was specifically arguing that conceiving of humans as "next token predictors" only works if you accept all the ramifications of that argument, which is the argument for hard determinism, aka free will is an illusion. And while a case could be made, I think such an argument is very unpalatable to the vast majority of people.
More options
Context Copy link
More options
Context Copy link
What should interest you here is how much of your supposed free will, and thoughts and processes and actions, are coming from the Default Mode Network.
Yes you may be someone with good mindfulness and a cultivated, strong agency and some original plans and actions every day. But - if you're like most people, you'll often be falling for this phenomenon: your mind wipes out a memory. For the thousandth time, it presents a thought process. You feel it's happening for the first time. You identify with it and carry it out as if this is free will etc. Then tomorrow or next week the exact same thing happens and you're fooled into believing you're experiencing it for the first time again and not just running a pattern.
uhhh I'm not following this at all, can you explain it more? The wikipedia article is strictly anatomical/medical and doesn't seem to have anything related to memory wiping?
I doubt you'll ever find this on wikipedia.
You'll either have to dig deep in the DMN literature/research, or try reading Mahasi Sayadaw's Manual of Insight or his The Progress of Insight.
Or start a serious meditation practice yourself. The phenomenon is well-known among adept meditators. A stronger, more detached awareness is built that can observe some of the mind's trickery that usually goes unnoticed.
I wish I had the time, I do love esoteric knowledge. I am unfortunately already consumed by the practical mundanities of life and the constrains they force upon you. I small outlet is technically//professionally relevant ML/AI esoterism around information utilization and learning knowledge.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
We evolved to have a constant experience-reaction feedback loop. If a bee stings me, the signal takes time to travel up my arm, get to my brain, and then be processed into action on my part.
If we imagine alien anthropologists who move and react to things in femtoseconds, they might look at humans staying still for eons as a bee lands on them, stings them, and then just let the bee stay there for millions of femtoseconds before slapping it and conclude that we're not truly conscious. We're constantly "starting and stopping" actions when nothing is going on, even if we have a relatively continuous, rolling awareness.
I don't see how it is that different from LLM's in principle, except that because we're designing them, we have to be the ones to put them into an agentic loop to accomplish things.
This essentially is the hard-determinism stance. You have no free will because you are just a causal mechanism probabilistically responding to external system according to your internal set of parameters (that you didn't choose). I don't think it really takes an alien to see that.
I believe humans can walk untrodden ground, that we have the ability to do things that are not causal mechanistically related to external stimuli in a way that an LLM currently does not. If you want to profer that you are just a flesh-bag robot with no free will, that is a belief system, but I'm not sure you'd like the ramifications of essentially being an object.
My feeling has always been that free will of this style is undesirable.
Consider something I do every day, like drive to and from work. I want my actions to be causally determined by my character, my memories and experiences and the kind of person I am, because I would never just decide to randomly swerve my car and hit the concrete barrier between the lanes at max speed.
But if I have the kind of radical free will that you propose, then there's always a possibility that, in spite of my upbringing, and the moral character I have spent my whole life cultivating and inhabiting, I could just make the random decision, causally unburdened by anything that has come before, to slam into the concrete divider head on at max speed in my car. I don't want the free will to "walk untrodden ground" that you propose. In a very real sense, it seems to me that whatever a-causal "decider" there is in me in such a situation, must not be me, since I would never have chosen to do the things a truly free version of myself would have chosen.
On the other hand, if I inhabit a deterministic universe, then I at least can know that whatever I do, it will be causally downstream of the person that I am, and that is comforting, regardless of whatever my ultimate fate will be. At least, on some level, I can say that I am truly the agent acting in the world, and reaping the consequences of my actions.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'm not arguing that humans are rational actors, but arguing that our cognition itself is largely based on something comparable to "next token prediction" is very much not established. Yes, humans recognize speech patterns and react to them, but those are only a small part of the working models our minds build of the world, our place in it, etc. and it is by no means clear that this works the same way as an LLM predicting tokens.
Sorry for the late reply. Been busy, but circling back to this has been on my todo list.
I mentioned Scott Adams because before LLMs were even a thing, his world view was basically that people don't think at all. They are hypnotized by language. They think in words, and those words can be used to completely hijack their consciousness, hypnotizing them. In his world view, people were only barely capable of a defense against this at best. And even at people's best, because our entire world model is often built on language, it ends up with all sorts of terrible pathways being laid down that are horribly wrong because of words and how lossy an abstraction they can be. Choose the right name for something, and even with firsthand experience telling them otherwise, people will walk away believing the opposite of reality.
All the shortcomings of human cognition that Scott Adams pioneered broad public awareness of pre-LLMs sound a whole awful lot like all the ways LLMs are shit these days. I never really followed up much with Scott Adams the last few years of his life. I don't know if he ever chased down this avenue too, but I hope he did.
I'll be deeply curious if LLMs can be trained to be better next token predictors than the humans that deny they are in the first place.
Well I don't think I have too much to say in response, but if you want a more in-depth exploration of the ideas you're talking about I'd recommend the anime film Genocidal Organ (or the novel it's based on, though I've only seen the film). It's all about the idea that language can be used and manipulated to control people in just about every single way. I think I mostly disagree with the arguments made in the film, but it presented some interesting ideas to chew on. It also has some good action scenes, including an orbital drop assault (probably the best part of the film).
Also Hideo Kojima was good friends with the author, and you can definitely tell that the vocal chord parasites in MGS5 were inspired by Genocidal Organ.
Thanks for the recommendation. I loved the antagonist's motivation.
For sure, glad someone found it useful! I found out about it originally when the orbital drop assault scene got recommended to me on YouTube a few years ago.
I think my next favorite part, mostly just because of how ridiculous it is, is when the antagonist has the protagonist captured and is monologing a bit before he plans to kill him and the protagonist shouts out "But the Sapir-Whorf hypothesis is false!" or something to that effect and that surprises and mildly amuses the antagonist. Well that, plus the blatant Domino's product placement.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link