Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.
- 159
- 1
Jump in the discussion.
No email address required.
Notes -
Not my real name, just my nom de plume "self_made_human".
First, I took a very old story I'd written, the one about my grandpa and his pet tiger. Why that one? Well, I was already in the process of rewriting it, though I shared the very first version that won be an AAQC ages ago, on the subreddit. I asked it to identify the author of the work without access to internet search.
It got it right the very first time. I was flabbergasted, and immediately tried 2 more times, and it failed. But out of 5 attempts, it guess self_made_human thrice in total, alongside other incorrect first guesses. It sometimes refused to guess at first, because of the risk of being incorrect, I told it that being wrong wasn't an issue at all, the whole point of the eval was to see how accurate the guesses were. And they were accurate.
In another chat, I asked it if it knew who self_made_human was. Once again, no web search. It guessed correctly on the first try in two instances, then claimed confusion the third one - yet when I prodded it to just go with whatever was on the tip of its tongue - it identified me and the topics I'd written on.
It struggled more on the third version of the experiment, where I used a more recent essay, but once again, light encouragement to guess let it get the right answer.
Pretty sure Claude couldn't do this before, and I do test on a semi-regular basis. Gemini 3.1 Pro very much can't, and it even cheated outright by searching after being told not to search (I don't think you can even turn off web search directly there). But the point is, a few paragraphs written ages ago, on the /r/TheMotte, which never was a massive sub, was enough to pin me down. And even newer material not in the training data was.
Is that recognizing you by style? Or is it just that the current training sets are so exhaustively scraped that even AAQC motte posts are included? I'm pretty sure they scraped reddit to the bone, right? Failing to connect the author of a unique reddit post literally in the training data 40% of the time actually sounds kind of horribly bad.
It was able to identify me from text written after the Jan 2026 knowledge cutoff.
Your expectations are far too high if you don't think this is impressive. Model weights are incredibly compressed in comparison to the training corpus, it's impressive when they remember moderately famous people, let alone someone like me who's only barely broken out. Associating an old Reddit post with my wider work and then accurately joining the dots is impressive. Do you see the average human seeing a random reddit comment from 5 years ago and then pinning it on the right person, and associating it with their other work? It's not even a post that went viral, even though it was an AAQC. It also independently associated much of my wider work with me, including posts on LW and RoyalRoad where I use a different username (even though I've linked between everything frequently enough). This is a clearly superhuman ability.
If that person had a searchable database? Sure.
It's possible I'm just entirely misunderstanding how they work down in the guts, but I interpreted the task as something like "I searched my database of training data, found the exact post, and replied with the linked username", which is powerful and "superhuman" in an objective sense, but the sort of thing I would have expected from Google search 15 years ago, pre-enshittification.
Identifying you by new writing would be much more impressive and alarming, and it sounds like they can actually do that for people like Scott, from some of the other posts people have made.
Models don't have access to training data at inference time.
Then, if it doesn't have access to internet search, how is it looking things up?
If I gave you a snippet of Shakespeare and asked you to guess who wrote it, I expect the Bard would be one of your top choices. How are you doing that if you don't consult Google or your Shakespeare box set?
Each token that the model sees in training updates its view of what sort of things are associated and in what way. Elements of style or topics may be clustered somewhere in the high dimensional latent space with the corresponding authors.
I would have similar examples in my memory. What does that memory look like for an LLM if not reference to a database? I just asked the Edge default copilot for "the 3rd line from Shakespeare's 31st sonnet", and for "without searching the web, the 5th line from Shakespeare's 41st sonnet" and it produced both without any trouble.
Are you suggesting that Copilot is re-deriving particular lines from the sonnets from first principles?
This is a sincere question, I honestly don't know how the nuts and bolts of these things work.
The memory is the weights of the model. The first stage of training an LLM is next token prediction - the LLM is shown a block of text and is trained to produce the next word (technically, part of a word, but that's not important). Internally, the model manipulates the numeric representation of the input tokens as points in a high dimensional space. The model produces a kind of probability distribution over all the words it knows for what the next word might be, and the weights of the model are adjusted so that it's more likely to expect the correct word.
The result is that every token, with some context before it, that the model sees leaves some kind of "impression" upon the model. The details are fuzzy, but things like word choice or style are probably represented in some regions of that high dimensional space, which is what lets the model say that something "sounds like" Shakespeare.
Anthropic have some public research around trying to look into a language model to see what's going on in that latent space and how neuron activations relate to concepts. It may be difficult to read without any ML background, but fortunately you can now feed it in to an LLM and have it explain anything you don't understand.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link