In this toy case it's just literally a calculator (a snippet of python code). The problem is 2+2, the calculator just does 2+2 and checks if the answer is the same as the LLM output. (The LLM is trained to format the final answer in a particular manner and wrap it with special tokens, so the verifier doesn't have to be able to interpret natural language.)
You can get surprisingly far with this. If it's a calculus question, you can use an automatic differentiator to check it. Likewise for factorisation questions, metric conversion questions, algebraic manipulation of formulae, etc. you put a little work into programming the automatic verifier and you can get an infinite number of problems.
If you're a big company, you might have human domain experts doing some of this work too. If you're a smaller company you have a big LLM do verification for the smaller ones.
Then you have leetcode and programming problems, and again you can verify these automatically. Does the program compile? Is the program output what was requested? Is it faster than the previous solution?
Like I said, this only works for maths, programming, and other domains where you can verify the answer with a computer relatively cheaply, but contra the model of multiple intelligence factors, heavy training on maths and programming seems to improve general intelligence and reasoning quite well.
Mid 30s, and I drink rarely because even one pint makes me woozy for a few hours and that's not fun unless I'm with friends. Drank 3 pints one evening last week, had very restless sleep and was hungover and unable to work until about 3pm. That's a bit extreme for me but it's just not something I can do any more.
In general I think it has less to do with age and more to do with drinking frequency, which correlates with age for various reasons. My father is like @MaximumCuddles and has more every single day than I would in a month. He doesn't sleep well but otherwise shows no ill effects.
Question: What is 2 + 2
Model: Hmm, that’s 2 and then another 2, so 22.
AUTOMATIC VERIFIER: WRONG
——
Model: Hmm, that’s the sum of 2 and 2, so 4
AUTOMATIC VERIFIER: CORRECT.
The model is tweaked slightly to make the second output more likely, and that output is potentially added to the training set. Repeat for arbitrarily complex mathematics and other problems as long as the solution can be verified, even if it isn’t known in advance. In this way you can generate potentially infinite amounts of data, albeit limited to certain domains. However, problem solving ability has so far extended quite well to other domains even when trained in this manner.
Your instinct is probably correct IMO. This form of synthetic data generation is just another tool in the box, it's not the key to everything.
I will say that we've got far further than I ever expected us to get using these methods. I'm instinctively a Gary Marcus-style fan of embodiment and unsupervised learning, it seemed clear to me pre-LLM that models wouldn't be able to be anything resembling intelligent without a body and the ability to interact with the real world and 'test' their understanding in real time. When LLMs came in, I felt I had to admit that I'd been wrong. It seems clear to me that we have managed to get to something I would call 'intelligence' (even if it's spiky and fails in some cases where humans would not fail) through these means. So I no longer trust my instincts as much.
This kind of semi-supervised exploration seems like a good compromise for now. I am also very interested in LLMs that can combine next-token video generation and text generation, because video generation requires understanding a bunch of stuff about the real world in order to produce consistent results, but that's a way off.
More options
Context Copy link