astrolabia
21hr ago
Can you help me understand this claim more concretely? E.g. if an LLM had just successfully designed a bridge for me, but then I modified the design to make it not useful in some way, for some kinds of changes it wouldn't be able to tell if my change was good or not? But a human would?
astrolabia
2d ago
I agree that alignment is easy in the sense of getting models to understand what we want, but it's far from clear that it's easy in the sense of making models want the same thing. RL models reward hack all the time.
What on earth makes you think instrumental convergence "doesn't actually happen"? It happens all the time, e.g. by reward hacking or sycophancy! It's almost the definition of agency!
Neuralese is a myth? What is that supposed to mean? RL on soft tokens is an active area of research and will almost certainly always work better (in the sense of getting higher rewards) than using hard tokens everywhere.
astrolabia
3d ago
none of it is going to happen in the way the AI safety movement predicts
Care to elaborate? What kinds of things do you think are going to happen differently than the AI safety people think?
astrolabia
26d ago
Is this a bit? Yes collecting a dataset is tons of work, but tokenizing it is trivial.
astrolabia
29d ago
I agree with everything you wrote in this reply. But your reply seems to have nothing to do with your message I originally replied to. Why were you mentioning the cost of tokenization?
astrolabia
29d ago
Why are you talking about the footprint of a tokenizer? Tokenization is cheap compared to actually evaluating the LLM.
It really is remarkable the strength of claims that otherwise smart people will make about the impossibility of AI doing something. As evidenced by IGI's reply, I think usually if someone has gotten this far without updating, you shouldn't expect a mere compilation of strong evidence to change their minds, but just to prompt the smallest possible retreat.
I had an amazing conversation with an academic economist that went along similar lines. I asked why his profession generally wasn't willing to even entertain the idea that AI could act as a substitute for human labor, and he said "well it's not happening yet, and making predictions is beyond the scope of our profession". Just mind-boggling.
To empathize a little, I think that people intuitively understand that admitting that a machine will be able to do everything important better than them permanently weakens their bargaining position. As someone who hopes humanity will successfully form a cartel before it's too late, I fear we're in a double-bind where acknowledging the problem we face makes it worse.
More options
Context Copy link