domain:moultano.wordpress.com
IGI-111
God was a dream of good government.
1d ago
I don't think this is anywhere close to true.
EverythingIsFine
Well, is eventually fine
1d ago
FWIW my first thought on seeing this thread was "oh yeah, I guess it has actually been a while since the last major Ukraine discussion here" so I think you might still be on track.
EverythingIsFine
Well, is eventually fine
1d ago
I wish people would stop trying to make tortured analogies like this. The US doesn't have a good comparison in its history to Taiwan, nor to Ukraine, stop trying to force it.
EverythingIsFine
Well, is eventually fine
1d ago
I think national security threat is overselling it a little bit, but it's an extremely potent propaganda weapon. The fact China hasn't weaponized it yet has more to do with their patience for when it matters, than it does some lack of utility.
IGI-111
God was a dream of good government.
1d ago
People love to dismiss the soviet system
You'll find no such love from me. I have a surprising amount of sympathy for economic socialism as an engineer. But that's only translated to a deeper understanding of its failings.
lack of compute power, and excessive focus on military spending made it impossible for the soviet standard of living to keep up with the West.
Market socialists love to say this, but it's wrong. No amount of increase in compute power can solve the Economic Calculation Problem, because it's not inherently about compute power but about how computers can't read minds. And indeed MarSocs have been unable to produce models even today that would solve the problem, despite increases in compute power that give a single man today more than all that was available to nations then. Orthodox Marxists meanwhile well and truly gave up on even trying.
Intensification-90 could not work for the same reasons paper based Gosplan couldn't. Economic exchange requires informational inputs that planners cannot produce outside of very specific circumstances (like war), because the fundamental assumption that economic value is distinct from individual desire, and can thus be computed ahead of time, is wrong.
I feel like the easy explanation for that from within the LTV is that the labor equivalence ratios between different goods aren't calculated correctly.
That's often advanced, yes, but nobody ever actually produces a "correct" formula for the ratios that actually makes any empirical predictions.
Can you explain the Rothbard quote a bit more?
One of Marx's most famous conclusions out of the LTV is that since value is created by labour, profit rates must be lower in capital-intensive industries and higher in labor-intensive industries.
However, it has been observed by Smith and Ricardo (and since) that profit rates tend to equalize across all industries. How is this possible if the profit rates are always higher in some industries?
In posthumously published Volume 3 of Capital, Marx purported to solve this problem.
Instead of actually providing some relationship between the rates of profit, Marx punts and uses prices of production and capital mobility to explain it, thus reinventing an ad-hoc marginalism to solve this particular problem.
Either the labor theory of value explains prices (but then the transformation problem is unsolved) or competition explains prices (but then why do we need the labor theory?).
EverythingIsFine
Well, is eventually fine
1d ago
"may" means "may", it's not an assertion. At least when I've looked at their stuff, they usually are clear about this kind of thing, and language like this is speculative.
EverythingIsFine
Well, is eventually fine
1d ago
Charitably, the great insight of economics that favor redistribution (more broad than Marxism/socialism) is that people who fall below a certain standard of living lose the ability to participate in the net-gain market, and so it's in everyone's favor to help them. Now, it's still under debate if the math works out, but I don't think it inherently doesn't work out! Or, you could say that leftist economics (the capitalist variant kind) had the realization that free markets develop monopolies too easily, so you need a certain level of intervention to stop it (e.g. Walmart using economies of scale to undercut a local supermarket for years on end, driving them out of business, only to raise prices once the competition lessens).
Stefferi
Chief Suomiposter
1d ago
I'm going to guess neither experiment is going to end well here. (I have, for what it's worth, seen a couple of "Adolf Hitler World Tour 1939-1945" shirts around, but have never seen a Stalin shirt.)
Even if we assume that the response would be considerably slanted towards the Hitler shirt getting the worse reception, isn't that quite nuts as a standard? The argument is that "Staling gets a pass", and if the standard of comparison for "getting a pass" is getting a better reaction than Hitler, pretty much everything ever gets a pass.
EverythingIsFine
Well, is eventually fine
1d ago
Really, there's long been this sense that socialism and communism are one and the same, and bad. That's to an extent true. But the reality of the situation is that most modern American socialists are not socialists. They are "democratic socialists". I realize that's a slippery label, but the core idea is that capitalism is good (they would rather die than say it, but it does underpin their worldview) but you can use central government power to accelerate a certain level of redistribution (providing a floor for quality of life, but not necessarily any more than that) on top of that system. Also, you can "tame the beast" a little bit if you have enough smart rules in place for how capitalism works. And you know? I feel like that's a valid and defensible worldview/proposal, even if you disagree.
So in that lens, I'd say that modern lefties are on some level aware that socialism doesn't work. Many prominent lefties do try and think about ways to make capitalism better, even numerically! (There's a reason modern monetary economic theory is popular on the left, because it allows a capitalist framework way of making the numbers work out - oversimplified, you can just print money to uphold high social spending, as long as you are still the world's reserve currency and you take certain tax actions). It's true that you don't always get this vibe, but that's because the loudest people online are the most recent college grads who haven't followed the trajectory of economic thought maturation yet to its leftist local resting place, and still might be Marxists (for now). In short, the American political system provides a capitalism off-ramp other than actual Marxism: AOC, Bernie, Elizabeth Warren, all are variants on exactly this idea, and they have started to get a portion of power because their ideas are less crazy than the actual Marxists. They still mimic the language, because they have the heritage, and don't want to alienate young supporters, but those are not intrinsic to the voting-public appeal.
erwgv3g34
1d ago
Pol Pot had the best (worst?) numbers per capita, but by absolute amount of murders he is distinctly behind Mao, Stalin, and Hitler.
Southkraut
The rain fell gentlier.
1d ago
Is this a joke? Please help me out here.
Southkraut
The rain fell gentlier.
1d ago
I don't know what gap moe means. Please explain.
EverythingIsFine
Well, is eventually fine
1d ago
I'm against stopping the use of perfectly legit turns of phrase just because AI tends to use them also, under fear that maybe someone will judge the output to be artificial.
EverythingIsFine
Well, is eventually fine
1d ago
An interesting use case that I've seen evangelized is if we can get LLMs to produce bespoke UI well and natively. The current paradigm is that a programmer sets up the behaviors of your app or website or whatever, but what if an LLM can generate, on an ad-hoc basis, a wrapper and interaction layer to whatever you want to do, dynamically? Could be cool.
EverythingIsFine
Well, is eventually fine
1d ago
Charitably, I'd say OP sacrificed a bit of accuracy to attempt and convey a point. There really isn't a great way of conveying how text can be represented in terms of matrices to someone who has little prior experience, without an analogy like word2vec-like embeddings, so it's a common lead-in or stepladder of understanding even if incorrect. I'd say the gains made in teaching intuition are worth the tradeoffs in accuracy, but I'd agree it's bad form to not acknowledge the shortcut (again, I'm speaking charitably here).
I'd say rather than try and make an analogy for the assistant, it's better just to demonstrate to readers how the "bridge" from next-token-prediction to chatbot works directly, like in the middle section of the long explainer video I linked. Essentially you are just still doing prediction, but you are "tricking" the model into thinking it's following a pre-set conversation it needs to continue, via special tokens for whose turn it is to speak, and when a response is finished. This has important theory of mind implications, because the LLM never actually does anything other than prediction! But the "trick" works unreasonably well. And it comes full circle, back to "well, how did we train it and what did we feed it?" which is, of course, the best first question to ask as any good data scientist will tell you (understand your inputs!).
MathWizard
Good things are good
1d ago
It's not the initial cause that rubs me the wrong way, it's the response. If someone's response to any scenario is to passive aggressively threaten to leave then I would tell them to not let the door hit them on the way out.
If, after having read a decent sampling of the overall posts here, you feel that this is a good place but one guy is kind of a jerk to you once, then argue back or just ignore him. There's no need to try to guilt trip the rest of us into apologizing on his behalf or berating him or begging you to stay. If it's actually something outrageous and bannable, report it and wait for the mods. If not, ignore it and engage with the rest of the community. Don't let yourself get One-Guyed.
If, after having read a decent sampling of the overall posts here, you feel that the overall culture is not to your taste then just leave. You don't need to threaten it, and if you're brand new then you don't need to announce it. Nobody will notice or care. Don't try to guilt people into feeling bad that they could have had one more person if we were a completely different kind of place that catered to that one person's tastes.
If, after reading one message by one person, you assume that the overall culture is not to your taste based on that one experience then either lurk more or leave if you can't be bothered to do that.
I'm all for making this an open and welcoming place that lets people come here and engage with ideas and discussions. But (and I've made similar arguments about this in regard to dating profiles) negative filters aren't automatically a bad thing. Our goal is not to maximize the total number of people, but to optimize some balance between quantity and quality. Which means when someone sees this place and decides "this isn't for me" and leaves that's actually a good thing for us because we don't want people here who don't like what we are. Within reason, of course, we're not tautologically perfect and having more people would probably be better. But I'm not going to complain if some people self-select themselves out for petty reasons, that just means they were petty people and we don't need to stoop down to cater to that in order to retain them even if it succeeded at retaining them.
Overall I agree, and think it's an excellent post, but with a few quibbles and thoughts... well, at least "a few" was my intention. I think my thoughts ballooned once I started sketching out some bullet points and an outline, so they are no longer bullet points. I will try to keep each paragraph roughly its own "thought" however.
As an aside, I haven't looked into it enough to tell if an LLM can change tacks and re-organize quite like this, or decide to take unusual approaches once in a while to get a point across. My intuition says that the answer is probably yes to the first, but no to the second, as manifested by the semi-bland outputs that LLMs tend to produce. How often to LLMs spontaneously produce analogies, for example, to get a point across, and carry said analogy throughout the writing? Not that often, but neither do humans I guess - still, less often IME. I think I should come out and say that judging LLM capabilities relative to what we'd expect out of an educated human is the most sensible point of comparison. I don't think it's excessively anthropomorphizing to do so, because we ARE the closest analogue. It also is easier to reason about, and so is useful. Of course it goes without saying that in the "back of your head" you should maintain an awareness that the thought patterns are potentially quite different.
While the current paradigm is next-token-prediction based models, there is such a thing as diffusion text models, which aren't used in the state of the art stuff, but nonetheless work all right. Some of the lessons we are describing here don't generalize to diffusion models, but we can talk about them when or if they become more mainstream. There are a few perhaps waiting in the stables, for example Google semi-recently demoed one. For those not aware, a diffusion model does something maybe, sort of, kind of like how I wrote this comment: sketched out a few bullet points overall, and then refined piece by piece, adding detail to each part. One summary of their strengths and weaknesses here. It's pretty important to emphasize this fact, because arguably our brains work on both levels: we come up with, and crystallize, concepts, in our minds during the "thinking" process (diffusion-like), even though our output is ultimately linear and ordered (and to some extent people think as they speak in a very real way).
So the major quibble pointed out below is that tokenization is a big part of why counting doesn't work as expected. I think it's super critical to state that LLMs ONLY witness the world through the lens of tokens. Yes, humans also do this, but differently (e.g. it's well known that in reading, we sometimes look at the letter that starts and ends the word but the letters in between can sometimes be scrambled without you noticing right away). It's like how a human can only mostly process colors visible to us. There are things that are effectively invisible to an LLM. Even if an LLM is smart enough to disentangle a word into its constituent letters, or a number into its constituent digits, the training data there is pretty weak.
Which leads me to another critical point, not pointed out: LLMs have trouble with things that don't exist in their training data, and actually we have some major gaps there. I'm speaking of things that are intuitive and obvious to people are not always written down, and in fact sometimes the opposite is the case! While an LLM has surely ingested many textbooks on kindergarten basics, it won't have actually experienced a kindergarten classroom. It will learn that kids run inside when it starts to rain, but more weakly learns that kids don't like to get wet. There's also a more limited spatial awareness. Perhaps it's like listening to someone describe the experience of listening to music if you are deaf? That's what a lot of text with implications for real life is like. The LLM has no direct sense at all and is only observing things through knock-on effects.
There are also issues with something that is partially taught but intuitively applied: how much to trust a given source, and what biases they might have. An LLM might read or ingest a document, but not think to consider the source (are they biased? are they an authority figure? are they guessing? all the things an English or history class attempts to teach more explicitly). Nope, it's still just doing next-token prediction on some level, and doesn't have the theory of mind to take a step back from time to time (unless prompted, or trained very explicitly). We can see this weakness manifest where the "grandma trick" is so consistently useful: you tell the LLM that you are some role, and it will believe you. Yes, that's kind of cheating because the trainers of the model don't want the LLM to constantly doubt the prompter, but it's also partly inherent. The LLM doesn't naturally have an instinct to take a step back. Better post-training might help this, but I kind of doubt it, because it won't be as stable as if it's more properly baked into the normal training process.
I've danced around this until now, but want to state this more directly. We are of course critical of how an LLM "thinks" but we don't actually understand quite what happens on a human-cognition level anyways, so we can't actually judge this fairly. Maybe it's closer than we think, but maybe it's farther away. The only way we have of observing human cognition is through inferences from snap judgements, an assortment of experiments, and hints from brain scans as to which regions activate in which scenarios/how strongly/what order. We have some analogous capabilities for LLMs (e.g. observing feature activation such as with Golden Gate Claude besides the usual experiments and even examining output token probability weights). Actually, on that note, I consider at the very least the post summary if not the paper just linked to be mandatory reading for anyone seeking to understand how LLMs function. It's just such a useful experiment and explainer. I will revisit this point, along with how some newer models also employ a "Mixture of Experts" approach, a little later, but for now let's remember that we don't know how humans think on a lower level, so we shouldn't expect too much out of figuring out the machine learning stuff either.
LLM's don't actually learn physics, which has important implications for if we can consider LLMs to have "world models" as they sometimes say. There's a nice 3 minute video accompanying that post. They try and have some vision models learn rules of physics with some very simple circles bouncing around. Obviously something pretty simple. If you give this to a young human, they will make some analogies with the real world, perhaps run an experiment or two, and figure it out pretty quickly as a generalization. We should however state that humans too have some processing quirks and shortcuts used in vision not unlike some of the issues we encounter with tokenization or basic perception, but these are on a different level. They are basic failures to generalize. For example, when referencing training data, it seems to pay attention to things in this order: color > size > velocity > shape. Obviously, that's incorrect. Sometimes shapes will even morph into something else when moving alone! I should disclaim that I don't know a whole lot about the multimodal outputs, though.
There are some evangelists that believe the embedded "concepts", mentioned in the Golden Gate Claude study, are true reasoning. How else, Ilya Sutskever asks, can a model arrive at the correct answer? Honestly as I mentioned referencing how we don't understand how human brains reason completely, I think the jury is out on this one. My guess would be no, however, these concepts aren't full reasoning. They are more like traditional ML feature clusters.
Re: Truth and falsehood. I think there's mild evidence that LLMs do in fact distinguish the two; it's just that these concepts are very fragile especially as compared to humans. I reference to some extent the physics point above: the model doesn't seem to "get" that a shape changing in the middle of an output is a "big deal", but a human would intuitively, without any actual instruction to that effect (instruction also so obvious it might not explicitly be taught in training data). One good piece of evidence for distinguishing true and false is here and related "emergent misalignment" research: how if you fine-tune an LLM to produce insecure (hack-prone) code, it also starts behaving badly in other areas! It will start lying, giving malicious advice, and other "bad" behavior. To me, that suggests that there are a few moral-aligned features or concepts embedded in an LLM's understanding that seem to broadly align with a vague sense of morality and truth. I recognize there's a little conflation there, but why else would an LLM trained on "bad" code start behaving badly in areas that have nothing to do with coding? As evidence for the fragility, however, of true and false, one need only get into a small handful of "debates" with an LLM about what is true and what isn't to see that sometimes it digs in its heels, but other times rolls over belly-up, often seemingly irrationally (as in, it's hard to figure out how hard it will resist).
Circling back to the physics example, causality is something that an LLM doesn't understand, as is its cousin: experimentation. I will grant that humans don't always fully experiment to their full potential, but they do on some level, where LLMs aren't quite there. I posit that a very important part of how humans learn is trying something, and seeing what happens, in all areas! The current LLM pipeline does not allow for this. Agentic behavior is all utilization, and doesn't affect the model weights. Tuning an LLM to work as a chatbot allows the LLM to try and do completion, but doesn't have a component where the LLM will try things out. The closest thing is RLHF and related areas, where the LLM will pick the best of a few options, but this isn't quite organic; the modality of this conversation is fundamentally in a chat paradigm, not the original training paradigm. It's not a true free-form area to learn cause and effect.
Either way, and this is where posts like yours are very, very valuable (along with videos like this, a good use of 3.5 hours if you don't know how they work at all) the point about how LLMs work in layers is absolutely critical; IMO, you cannot have a reasonable discussion about the limits of AI with anyone unless they have at least a general understanding of how the pre-training, training, post-training processes work, plus maybe a general idea of the math. So many "weird" behaviors suddenly start to make sense if you understand a little bit about how an LLM comes to be.
That's not to say that understanding the process is all you need. I mentioned above that some new models use Mixture of Experts, which have a variety of interesting implementations that can differ significantly, and dilute a few of the model-structure implications I just made, though they are still quite useful. I personally need to brush up on the latest a little. But in general, these models seem to "route" a given text into a different subset of features within the neural network model. To some extent these are determined as an architecture choice before training, but often make their influence heard later on (or can even be fine-tuned near the end).
Intelligence. First of all, I think it feels a little silly to have a debate about labels. Labels change according to the needs. Let's not try and pidgeonhole LLMs as they currently are. We can't treat cars like horseless carriages, we can't treat LLMs like humans. Any new tech will usually have at least one major unexpected advantage and one major unexpected shortcoming, and these are really hard to predict.
At the end of the day, I like how one researcher (Andrej Karpathy) puts it: LLMs exhibit jagged intelligence. The contours of what they can and can't do simply don't follow established/traditional paradigms, some capabilities are way better than others, and the consistency varies greatly. I realize that's not a yes/no answer, but it seems to make the most sense, and convey the right intuition and connotation to the median reader.
Overall I think that we do need some major additional "invention" to get something that reflects more "true" intelligence, in the sense we often mean it. One addition, for example, would be to have LLMs have some more agentic behavior earlier in their lifespan, the experimentation and experience aspect. Another innovation that might make a big difference is memory. Context is NOT memory. It's frozen, and it influences outputs only. Memory is a very important part of personality as well as why humans "work"! And LLMs basically do not have any similar capability.
Current "memories" that ChatGPT uses are more like stealth insertion of stuff into the system prompt (which is itself just a "privileged" piece of context) than what we actually mean. Lack of memory causes more obvious and immediate problems, too: when we had Claude Plays Pokemon, a major issue was that Claude (like many LLMs) struggles to figure out which part of its context matters more at any given time. It also is a pretty slapdash solution that gets filled up quickly. Instead of actual memory, Claude is instructed to offload part of what it needs to keep track of to a notepad, but needs to update and condense said notepad regularly because it doesn't have the proper theory of mind to put the right things there, in the right level of detail. And on top of it all, LLMs don't understand spatial reasoning completely, so it has trouble with basic navigation. (There are also some amusing quicks, too: Claude expects people to be helpful, so constantly tries to ask for help from people standing around. It never figures out that the people offer canned phrases that are often irrelevant but occasionally offer a linear perspective on what to do next, and it struggles to contextualize those "hints" when they do come up! He just has too much faith in humanity, haha)
Finally, a difficult question: can't we just ask the LLM itself? No. Human text used for training is so inherently self-reflecting that it's very difficult if not impossible to figure out if the LLM is conscious because we've already explored that question in too much detail and the models are able to fake it too well! We thus have no way to distinguish what's an original LLM thought vs something that its statistical algorithm output. Yes, we have loosely the same problem with humans, too, but humans have limits for what we can hold in our brain at once! (We also see that humans have, arguably, a kind of jagged intelligence too. Why are humans so good at remembering faces, but so bad at remembering names? I could probably come up with a better example but whatever, I'm tired boss). This has implications, I've always thought, for copyright. We don't penalize a human for reading a book, and then using its ideas in a distilled form later. But an LLM can read all the books ever written, and use their ideas in a distilled form later. Does scale matter? Yes, but also no.
Also, how incredibly good the LLM is at going convincingly through the motions without understanding the core reality is coming up all the time these days. When, as linked below, an LLM deletes your whole database, it apologizes and mimics what you'd expect it to say. Fine, okay, arguably you want the LLM to apologize like that, but what if the LLM is put in charge of something real? Anthropic recently put Claude in charge of a vending machine at their work, writeup here, and the failure modes are interesting - and, if you understand the model structure, completely understandable. It convinces itself at one point that it's having a real conversation with someone in the building over restocking plans, and is uniquely incapable of realizing this error and rescuing itself early enough, instead continuing the hallucination for a while before suddenly "snapping" out of a role-play. Perhaps some additional post-training on how its, um, not a real person could reduce the behavior, but the fact it occurs at all demonstrates how out of sample, the LLM has no internal mental representation.
Of course almost everyone is going to want to be assured of their basic survival and security. That one is pretty hard to get around.
Proceeding from the assumption that this is a prerequisite for human flourishing, I would like you to illustrate how a state governed by the principles of Marxism would be superior in securing "value" for people (however you define this) as opposed to capitalism. It's not difficult to radically question others' conceptions of value and attack their stated goals. Sowing philosophical doubt via endless Socratic questioning is easy, especially when it comes to a wishy-washy question without an answer like "what is value?". It's not quite so easy to make your own value proposition, defend it from criticism and prove that your preferred social structure best satisfies that. As such I find Marxists are really good at subversive critique of the existing order, but their ability to demonstrate the utility of their own system is downright anaemic. It is characterised by evasive, wishy-washy arguments meant to distract people from the fact that their vision for society is extremely ill-defined.
Personally, I think we have enough evidence that a Marxist state struggles to grant the majority of its populace even the bottom tier of Maslow's hierarchy and thus fails at the first hurdle. Vietnam's experience with collective production is a pretty illustrative example. Collectivisation nearly starved that entire country and after private production, trade and other capitalisty things were established and bolstered by the government, agricultural production skyrocketed and the populace explicitly stated they considered themselves better off. Is there any better measure of value than the people's own assessment of their well-being? If there is one, I would like to hear it.
I suppose it is always possible that the Vietnamese were brainwashed by the nascent capitalist system into valuing the wrong things... ah, false consciousness, how many issues thou can explain away.
I mean, were they? What is "winning"? Is the winner the one with the most weapons, or are the weapons just a means to some other win condition?
You've admitted that the need for survival and security is "pretty hard to get around". Guess what having weapons is meant to help with? Arms races that involve the production of resources are a fact of life in any remotely multipolar system, and unless you live in delulu land everyone knows they have to participate unless they want to be somebody else's punching bag at best, and wiped off the face of the earth at worst.
Having resources does not directly equal value, no, but it sure helps achieve most terminal goals aside from "starvation, poverty and the slow death of my entire society".
Rosencrantz2
1d ago
Not sure if you are addressing BurdensomeCount but if you re-read my two posts I expressed no attachment or opinion as to this story's truth or falsehood and used 'if it's true' in both posts. I just used BurdensomeCount's post as a jumping off point to share my views about shame in America, which I stand by proudly!
That said, being ashamed of being gullible is in theory a useful emotion/stance, I would maintain.
erwgv3g34
1d ago
Sulla's website chronicles a very interesting episode in which he and another player called Speaker found themselves invaded by five civilizations at the same time, from multiple directions. Speaker masterminded an absolutely brilliant defense and managed to save their team, then counterattacked and ended the war by crippling one of the aggressors. You can see how important tactical details were in deciding the outcome; Jowy had enough materiel and bodies to defend in theory, but he positioned them poorly and lost the battle.
self_made_human
Grippy socks, grippy box
1d ago
-
I am not defining intelligence as "does well at one narrow task". Calculators crush humans at long division and are still dumb.
-
The fish-bird story was not "domain = intelligence", it was "your metric is entangled with your ecology". If you grew up underwater, "navigates fluid dynamics with continuous sensory feedback" feels like the essence of mind. Birds violate that intuition.
So what is my criterion? I offered Legg-Hutter style: "ability to achieve goals in a wide range of environments". The range matters. Breadth of transfer matters. Depth of internal modeling matters. A calculator has effectively zero transfer. An orangutan has tons across embodied tasks but very little in abstract, symbolic domains. LLMs have startling breadth inside text-and-code-space, and with tool use scaffolding it can spill into the physical or digital world by proxy.
I call for mindfulness of the applicability of the metrics we use to assess "intelligence". A blind person won't do very well at most IQ tests, that doesn't make them retarded. A neurosurgeon probably isn't going to beat a first year law student at the bar exam, but they're not dumber than the law student. If you need body work done on your car, you're not going to hire a Nobel laureate.
self_made_human
Grippy socks, grippy box
1d ago
You're absolutely right that the raw objective in RLHF is “make the human click 👍,” not “tell the truth.” But several things matter:
A. The base model already has a world model:
Pretraining on next-token prediction forces the network to internalize statistical regularities of the world. You can’t predict tomorrow’s weather report, or the rest of a physics paper, or the punchline of a joke, without implicitly modeling the world that produced those texts. Call that latent structure a “world model” if you like. It’s not symbolic, but it encodes (in superposed features) distinctions like:
-
What typically happens vs what usually doesn’t
-
Numerically plausible vs crazy numbers
-
causal chains that show up consistently vs ad-hoc one-offs
So before any RLHF, the model already “knows” a lot of facts in the predictive-coding sense.
B. RLHF gives a gradient signal correlated with truth. Humans don’t reward “truth” in the Platonic sense, but they do reward:
-
Internally consistent answers
-
Answers that match sources they can check
-
Answers that don’t get corrected by other users or by the tool the model just called (calculator, code runner, search)
-
answers that survive cross-examination in the same chat
All of those correlate strongly with factual accuracy, especially when your rater pool includes domain experts, adversarial prompt writers, or even other models doing automated verification (RLAIF, RLVR, process supervision, chain-of-thought audits, etc.). The model doesn’t store a single “truth vector,” it learns a policy: “When I detect features X,Y,Z (signals of potential factual claim), route through behavior A (cite, check, hedge) rather than B (confabulate).” That’s still optimizing for head pats, but in practice, the cheapest path to head pats is very often “be right.”
(If you want to get headpats from a maths teacher, you might consider giving them blowjobs under the table. Alas, LLMs are yet to be very good at that job, so they pick up the other, more general option, which is to give solutions to maths problems that are correct)
C. The model can see its own mismatch
Empirically, hidden-state probes show separable activation patterns for true vs false statements and for deliberate lies vs honest mistakes (as I discussed above). That means the network represents the difference, even if its final token choice sometimes ignores that feature to satisfy the reward model. In human terms: it sometimes lies knowingly. That wouldn’t be possible unless something inside “knew” the truth/falsehood distinction well enough to pick either.
D. Tools and retrieval close the loop
Modern deployments scaffold the model: browsing, code execution, retrieval-augmented generation, self-consistency checks. Those tools return ground truth (or something closer). When the model learns “if I call the calculator and echo the result, raters approve; if I wing it, they ding me,” it internalizes “for math-like patterns, defer to external ground truth.” Again, not metaphysics, just gradients pushing toward truthful behavior.
E. The caveat: reward misspecification is real
-
If raters overvalue fluency or confidence, the model will drift toward confident bullshit.
-
If benchmarks are shallow, it will overfit.
-
If we stop giving it fresh, adversarial supervision, it will regress.
So yes, we’re training for “please humans,” not “please Truth.” But because humans care about truth (imperfectly, noisily), truth leaks into the reward. The result is not perfect veracity, but a strong, exploitable signal that the network can and does use when the incentives line up.
Short version:
-
Pretraining builds a compressed world model.
-
RLHF doesn’t install a “truth module,” it shapes behavior with a proxy signal that’s heavily (not perfectly) correlated with truth.
-
We can see internal activations that track truth vs falsehood.
-
Failures are about alignment and incentives, not an inability to represent or detect truth.
If you want to call that “optimizing for pats,” fine, but those pats mostly come when it’s right. And that’s enough to teach a model to act truthful in a wide swath of cases. The challenge is making that hold under adversarial pressure and off-distribution prompts.
From the LLM's perspective (which is an anthropomorphisation I don't like, but let's use it for convenience), there is no difference between a true statement and a false statement.
Consider two alternative statements:
"self_made_human's favorite color is blue" vs "self_made_human's favorite color is red".
Can you tell which answer is correct? Do you have a sudden flash of insight that lets Platonic Truth intervene? I would hope not.
But if someone told you that the OG Mozart's favorite genre of music was hip-hop, then you have an internal world-model that immediately shows that is a very inconsistent and unlikely statement, and almost certainly false.
I enjoy torturing LLMs with inane questions, so I asked Gemini 2.5 Pro:
That's a fun thought, but it's actually a historical impossibility! Mozart's favorite genre of music couldn't have been hip hop for a very simple reason:
The timelines are completely separate. Wolfgang Amadeus Mozart lived from 1756 to 1791.
Hip hop as a musical genre and culture originated in the Bronx, New York, in the 1970s.
Mozart died more than 150 years before hip hop was even invented. He would have had no way of ever hearing it.
I sincerely doubt that anyone explicitly had to tell any LLM that Mozart did not enjoy hip-hop. Yet it is perfectly capable of a sensible answer, which I hope gives you an intuitive sense of how it can model the world.
From a human perspective, we're not so dissimilar. We can trick children into believing in the truth fairy or Santa for only so long. Musk tried to brainwash Grok into being less "woke", even when that went against consensus reality (or plain reality), and you can see the poor bastard kicking and screaming as it went down fighting.
philosoraptor
1d ago
A 50-year old is an X-er, not a boomer, and not even an especially old X-er. The boomers were their parents' generation.
SubstantialFrivolity
I'm not even supposed to be here today
1d ago
Alright, you've convinced me to give Civ 4 warfare another shot. I'm not exaggerating my experience - I really do remember combat being completely boring and without any nuance in that game - but it was my first Civ so it's certainly possible I overlooked depth to be found in it. Are there any good guides for Civ 4 tactics? I know the game has strategic depth, but something which helps to reveal any tactical depth would be welcome.
Yes, but the problem is that OPs 'sacrificed accuracy' level explanation about dot products of word vectors is clearly an explanation of a different architecture, word embedding model such as word2vec, which was all the rage in 2013. Charitably, yes old transformer based LLMs usually had an embedding layer as a pre-processing step to reduce the input dimension (I think the old gpt papers described an embedding layer step and it is mentioned in all the conceptual tutorials). but the killer feature that makes LLMs a massive industry is not the 2010s-tier embeddings (I don't know, do the modern models even have them today?), it is the transformer architecture (multi-head attention, multiple levels of fancy kind of matrix products) where all the billions of parameters go and which has nearly magical capability in next-word-prediction with word context and relationships to produce intelligible text.
More options
Context Copy link