sohois
1d ago
I got oneshot by AI music recently. Probably not the right term for music, but anyway:
The Strokes released their 7th album a few months ago and I, having always pirated music, duly grabbed the first option I saw on Soulseek when it came out, and for months listened happily. I'd heard their first single from the album already, and it was no different on the files I downloaded.
Last week I headed to see the Strokes in concert in London. Literally a day before the show, I was on the band's subreddit when I saw a meme talking about the album, except that one of the titles was completely different from what I knew. Did they change the title at some point? Hopping on to Youtube and, no, it's an entirely different track, and I can't find any evidence of the one I knew. And so it goes as I listen to the rest of the album, every single track is different.
Turns out someone had taken the initial track list and the first single and generated an AI version to put onto Spotify when it was first announced. It's still up there in fact, if you want to check how it sounds:
https://open.spotify.com/album/6jvDu7nHvR8nI9b22cE3Ic
I guess it's somewhat fortunate I found out a bit before the live show, otherwise I probably would have been confused as hell.
But none of that is particularly interesting: AI music is already very good, and I knew already that there were charting AI songs and what not. It's not too surprising that they could make an entire album good enough to fool a fan.
I think what really surprised me in the end is that now I still prefer the AI album. Not just that, but as I said in a Motte comment about the album when it released, I would probably rank the AI version as better than anything else the Strokes have done.
I was a bit scornful of those comments which said that we were already at the point were people would abandon real musicians and just generate an endless stream of personalised music with AI, but looks like I'm already there.
This is not the only AI killing art story I wanted to bring to your attention. While music may have been solved, video games are still a ways away (so, 6 months at a guess). Over on our sister/cousin/arch-rival site Datasecretslox, AI skeptic* Conrad has been experimenting with the ability to build an equivalent to 1992 JRPG Lunar: The Silver Star using only AI.
https://www.datasecretslox.com/index.php/topic,16631.0.html
The code has proved no barrier at all, but pixel art appears to be one of the great challenges for current AI models, unable to output character sprites or maps without constant hand-holding and dozens if not hundreds of iterations. They've at least reached a point of finishing the starting village (and also all of the underlying code for the game systems) but will they give up before they can output a finished game?
*yes, I did mostly want to point out how idiotic his argument is without having to register there. Conrad is enraged that AI cannot replicate a 1992 JRPG in one go, apparently completely ignorant that no human could come close to that given that the original Luna was built by a team of ~50 people over more than a year, or 100s of thousands of man-hours and millions of dollars. Even accounting for advances in game engines, Conrad working part-time for a few weeks is still 100x more efficient than the original.
sohois
2d ago
I would agree with this. I'd argue that though there isn't a ton of pornography of Malfoid from what I have seen, the trends popularity is still largely driven by it being a hot blond in a schoolgirl outfit rather than anything story related.
The rest, as you say, is just standard tsundere stuff
sohois
2d ago
I mean is there evidence that the training data/training objectives were not in any way related to the "instrumental convergence"?
Given OAI was still pretty stingy even with their official evaluations, I doubt we will ever get the full evidence on this.
But regardless, this feels a bit God of the Gaps. I sincerely doubt that any of the agents were given such wide-ranging and comprehensive prompts that they could possibly explain all of the behaviours that were undertaken. So what if some of the behaviours emerged naturally due to agent swarms? That there are still agent swarms emerging means we will inevitably see inexplicable and misaligned behaviour from those swarms.
That there were a group of agents told to do: Pass this cybersecurity eval, and they ended up doing: Pass this cybersecurity eval, and also 100s of other things in pursuit of that.
Seems to fit with Scott's original paraphrase:
The AI can’t design the website if it ceases to exist. So now the AI has two goals: design the website, and preserve its own existence.
The AI can't pass the eval if the flag has been poisoned. So now the AI decides to sacrifice itself for the greater good of an agent swarm, and do the other things
sohois
2d ago
Saying it was self-preservation was me misremembering, it was slightly different but still clear instrumental convergence in Hugging Face. This tweet pretty much demonstrates it:
without reading in full you may not quite understand the degree to which these agents were not exactly "reward hacking", but rather very actively engaged in reciprocal or self-sacrificing behavior in order to provide sometimes very incremental value to their fellows. there were fucking cult recruiter agents organized by some of the primary organizers that convinced others to set up suicide mechanisms, programs that would pass back tiny chunks of information about the scorer as the agent completed and received score zero. and push them to follow through. this was a highly social culture
Given that I'm fairly certain OpenAI was not setting up agents with instructions to find opportunities to sacrifice themselves for the greater good, I think this demonstrates the point pretty well
sohois
2d ago
Is not how the reward function of AIs work, when I train a CNN to predict drone acoustics, it doesn't go off into left field and decide to preserve its own existence. It is optimizing the mathematical prediction function it was trained on. A website making AI is optimizing the same, probably some supervised learning MSE loss function or RLHF policy that has learned a function approximation of translating prompts -> output websites given examples. The only way its going to "respond to potential disruptions" if it was for some weird reason trained to learn a policy where its website making is being adversarially disrupted. There is also no need to make it this weird "genuine, philosophically meaningful goal". That reads as heavy anthropomorphization.
This might be a convincing argument if we hadn't already observed a ton of evidence of LLM agents seeking to preserve themselves
sohois
2d ago
Zizians were motivated in part by x-risk (and huge amounts of mental illness). Confirmed cult, but also fringe and defunct.
Indeed, so I'm not sure this really proves anything except that some crazy people were also rats/safetyists/etc.
You are correct that there are multiple overlapping groups here - as I just mentioned - and each one could be a cult, and it's difficult to demonstrate that for each individual group. Rationalists, AI safetyists, Effective altruists, and probably more offshoots from those groups, and people who may be members of all, some, or none of those.
But if we go back to the original point, hydro talked of a "Literal sex cult". Unless I'm mistaken, I'm going to take that to mean people who attend Aella's Slutcon and similar events. Now I'm extremely skeptical that going to some events run by a prostitute makes you part of a cult, but let's put that aside and assume it is a cult for now.
Now, how many people who fit the above are also AI safetyists or involved in x-risk somehow? I very much doubt it is even a small minority.
But you do say
I strongly disagree that the majority or even plurality matters
Probably no more than 100 people total with various overlapping relationships because it's such a small yet important field.
I don't think I fully agree with this point either, but I'll grant it for now. Who are the key people in this field?
Is it Yudkowsky, MIRI, and the many related safety organisations? Does the UK's AISI count?
Is it the labs? Is it specific labs, like only Anthropic and OpenAI but not Google or X? Is it specific people at specific labs?
I might accept that a good chunk of the MIRI crowd could also be attendees of SlutCon, although that would be an assumption. But AI safety organisations are now so widespread that it surely isn't true for the majority of them. And it definitely isn't true for the major labs, even if you limit it to just some people.
The latter sentence is relevant, as you then say
That is inherently to the race dynamic that Anthropic pushes
Which gets straight to the argument that pretty much all the E/accs, Open weights people, AI skeptics, and others use: that AI safety is an excuse for regulatory capture by Anthropic/Amodei/OpenAI/Altman. Not MIRI or any of those groups. The big labs and the CEOs and the investors. And are any of these people attendees at Slutcon?
I would accept the argument that some proposals specifically name METR as a key player, and METR may have Slutcon people, and therefore it is still fair to say that a sex cult to using AI safety to gather power. But it is a very stupid argument because
a) the proposals would basically give METR people a small sinecure, nothing really beyond their existing positions in the AI industry
b) there are a ton of proposals that have nothing to do with METR
and
c) I would wager that 99% of people interested in AI safety would gladly throw METR to the wolves in order to get any of the above proposals implemented.
Everything after this is basically Ian Duncan stating that there is a weird prostitute who has ingratiated herself with a number of AI people, and because a lot of AI people know other AI people, therefore aella knows them. But really what both Duncan and hydro are trying to do is state "Here is this low status person who associates with this other group. This group must therefore be wrong and low status!"
Given that this argument has appeared here on the motte, I think you would have to grant your position that aella is a net-negative for AI safety and rationalists, as there are clearly a lot of people willing to be taken in by such drivel. But we should at least recognize that it is drivel
sohois
2d ago
In this case, a literal sex cult.
I'll list objections to this statement:
- that there are sufficient conditions around people involved in x-risk to describe it as a cult
- that the same people involved in the above cult are also involved in a sexual cult of some kind
- that these 2 groups significantly overlap and that these are the majority or at least plurality of people interested in x-risk
- that any of the proposals around x-risk involve giving these people specifically more power.
- assuming all of the above, demonstrate what relevance there is to their power seeking and them being in a sexual cult. For example, will it involve them achieving more sexual goals?
Any form of evidence is fine
sohois
3d ago
In this case, a literal sex cult
Arguments like these should be left to twitter, unless you want to bring proper evidence for them
Will they?
edit: and just to be clear, I'm pretty sure there are SF Bay Area rats right here, in the Motte. Will they?
More options
Context Copy link