site banner
magic9mushroom  If you're going to downvote me, and nobody's already voiced your objection, please reply and tell me  8mo ago (thezvi.wordpress.com)   2036 thread views

Danger, AI Scientist, Danger

thezvi.wordpress.com

Zvi Mowshowitz reporting on an LLM exhibiting unprompted instrumental convergence. Figured this might be an update to some Mottizens.

10
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

they aren't good at synthesizing information from two fields in ways that haven't been done before

They are not good at that yet. But there are already indicators that they could become so.

  1. Midjourney will blend concepts from different places. Not just style transfer. A nice example I saw was an image of 'boy with a hedgehog'. The boy was holding a hedgehog, but also his hair was a bit spiky, like the hedgehogs. I think it is most unlikely MidJourney had ever seen a composition/juxtaposition of that kind before, and both the hedgehog and the hair were modified to make the composition work.
  2. DeepMind have AlphaZero, which plays chess, shogi and go. It plays better than human, i.e. not just based on play it has seen before, and one can argue it is crossing between different genres, not confined to one field.
  3. The often cited example of finding an analogy between compost heap and nuclear fission, again an example of crossing field boundaries.

So to say that machine learning can't synthesise information from two fields in ways that have not been done before needs more qualification, to be defensible.

I was talking about (transformer-based generative) LLMs specifically. I am not a sufficiently good mathematician to feel confident in this answer, but LLMs and diffusion models are very different in structure and training, and I don't think that you can generalise from one to the other. Midjourney is basically a diffusion model, unscrambling random noise to 'denoise' the image that it thinks is there. The body with spiky hair seems like the model alternatively interpreting the same blurry patch of pixels as 'spikes' because 'hedgehog' and 'hair' because 'boy'. Which I think is very different from a predictive LLM realising that concept A has implications when combined with concept B that generates previously unknown information C.

DeepMind have AlphaZero, which plays chess, shogi and go. It plays better than human, i.e. not just based on play it has seen before, and one can argue it is crossing between different genres, not confined to one field.

I haven't kept up to date on RL, but I don't think this is relevant. Firstly because the concept of self-play is not really relevant to text generation, and secondly because I don't suppose the ability to play chess is being applied to go. Indeed, I don't really see how it could be, because the state and action space is different for each game. It seems more likely to me that the same huge set of parameters can store state-action-reward correlations for multiply games simultaneously without that information interacting in any significant way.

The often cited example of finding an analogy between compost heap and nuclear fission, again an example of crossing field boundaries.

I'm not aware of this. Can you give some more info?

Diffusion models work for text too. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10909201/

The blending of concepts that we see in MidJourney is probably less to do with the diffusion per se as with CLIP - a building block within diffusion. CLIP aligns a language model with an image model. Moving concepts between different representations helps with concept generation. There's a lot being done with 'MultiModal models' to make the integration between different modalities work better.

'Self play' is relevant for text generation. There is a substantial cottage industry in using LLMs to evaluate the output of LLMs and learn from the feedback. It can be easier to evaluate whether text 'is good' than it is to generate good text. So multiple attempts and variations can lead to feedback and improvement. Mostly self play to improve LLMs is done at the level of optimising prompts. However the outputs improved by that method can be used as training examples, and so can be used to update the underlying weights.

https://topologychat.com is a commercial example of using LLMs in a way inspired by chess programming (Leela, Stockfish). It does a form of self play on inputs that have been given to it, building up and prioritising different lines. It then uses these results to update weights in a mixture of experts model.

Here's the quote from Geoffrey Hinton:

"why is a compost heap like an atom bomb? And GPT-4 says, well, the timescales and the energy scales are very different. That’s the first thing but the second thing is the idea of a chain reaction.

So in an atom bomb, the more neutrons around it, the more it produces, and in a compost heap, the hotter it gets, the faster it produces heat and GPT-4 understands that. My belief is when I first asked it that question, that wasn’t anywhere on the web. I searched, but it wasn’t anywhere on the web that I could find. It’s very good at seeing analogies because it has these features. What’s more, it knows thousands of times more than we do. So it’s gonna be able to see analogies between things in different fields that no one person had ever known before.

That may be this sort of 20 different phenomena in 20 different fields that all have something in common. GPT-4 will be able to see that and we won’t. It’s gonna be the same in medicine. If you have a family doctor who’s seen a hundred million patients, they’re gonna start noticing things that a normal family doctor won’t notice."

From transcript at https://medium.com/@jalilnkh/geoffrey-hinton-will-digital-intelligence-replace-biological-intelligence-fc23feb83cfb of the video.

Yes diffusion models work for text too. The difference between a collated set of pixels (ie an image) and a collated set of letters (ie a word or sentence) is purely conceptual. From an algorithmic perspective it's all just "tokens". However, this overlap in operation doesn't mean that they are not very different beasts under the proverbial hood.

By way of analogy, a conventional piston engine, a turbine engine, and an electric motor attached to a battery may all accomplish the base task of "make the vehicle go" but they have different trade-offs, use cases, and operating principles. Point being that similar output does not equal similar function.

As someone who has actually spent some time "in the trenches" as it were, designing algorithms and writing code to execute them, I am in broad agreement with @Corvos's take. The opening of Mowshowitz's essay comes across as ignorant, lazy, and plainly self-serving and nothing that follows really challenges that first impression of him.

The link from Hinton is better but seems make a lot of similar mistakes. It seems clear to me that both are far more interested in driving engagement through hyperbole than really exploring or helping others understand the underlying questions and theories.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats