This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Since @Hawaii98 complains about insufficient quantity of quality commentary, I've taken it upon myself to cover one of the topics proposed by @greyenlightenment, namely the doxxing of Based Beff Jesos, the founder of effective accelerationism. My additional commentary, shallow though it may be, got out of hand, so it's a standalone post now: E/acc and the political compass of AI war.
Quoting Forbes:
My main objective is to provide the reader with convenient links to do own research and contribute to the debate, so I rapidly switch from Beff to a brief review of new figures in AI safety discourse, and conclude that the more important «culture war» of the future will be largely fought by the following factions:
In the spirit of making peace with inevitability of most discussion taking place in the main thread, I repost this here.
edit: not to toot my own horn, but
I am checking here less and less often because A) with my current concerns and the way wind blows, Western culture war is largely irrelevant B) there's little for me to contribute in addition to all that has been said and C) I've concluded that my ability at making commentary is better used for making an impact.
edit 2: I also mildly dislike the fact that standalone posts need approval, though I can see how that follows from the problem/design choice of easy anon registration.
Repasting my own lengthy comment:
I've always been a techno-optimist (in the sense that I strongly believe that technology has been the biggest positive force for good in history, likely the only form of true progress that isn't just moral fashion), but these days I'd call myself d/acc instead of an e/acc, because I think current approaches to AGI have a subjective probability of about 30% of killing us all.
I don't call myself a doomer, I'd imagine Yud and co would assign something like 90% to that, but in terms of practical considerations? If you think something has a >10% of killing everyone, I find it hard to see how you could prioritize anything else! I believe Vitalik made a similar statement, one more reason for me to nod approvingly.
A large chunk of the decrease in my p(doom) from a peak of 70% in 2021 to 30% now is, as I've said before, because it seems like we're not in the "least convenient possible world" where it comes to AI alignment. LLMs, as moderated by RLHF and other techniques, almost want to be aligned, and are negligibly agentic unless you set them up to be that way. The majority of the probability mass left, at least to me, encompasses intentional misuse of weakly or strongly superhuman AI based off modest advances on the current SOTA (LLMs) or a paradigm shifting breakthrough that results in far more agentic and less pliable models.
Think "Government/Organization/Individuals ordering a powerful LLM to commit acts that get us all killed" versus it being inherently misaligned and doing it from intrinsic motivation, with the most obvious danger being biological warfare. Or it might not even be one that kills everyone, an organization using their technological edge to get rid of everyone who isn't in their in-group counts as far as I'm concerned.
Sadly, the timelines don't favor human cognitive enhancement, which I would happily accept in the interim before we can be more confident about making sure SAGI is (practically) provably safe. Maybe if we'd cloned Von Neumann by the ton a decade back. Even things like BCIs seem to have pretty much zero impact on aligning AI given plausible advances in 5-10 years.
I do think that it's pretty likely that, in a counterfactual world where AI never advances past GPT-4, ~baseline humans can still scale a lot of the tech tree to post-scarcity for matter and energy. Biological immortality, cognitive enhancement, interstellar exploration, building a Dyson Swarm or three, I think we could achieve most of that within the life expectancy of the majority of people reading this, especially mine. I'd certainly very much appreciate it if it all happened faster, of course, and AI remains the most promising route for that, shame about everything else.
I have no power to change anything, but at the very least I can enjoy the Golden Age of Humanity-as-we-know-it, be it because the future is going to be so bright we all gotta wear shades, or because we're all dead. I lean more towards the former, and not even because of the glare of nuclear warfare, but a 30% chance of me and everyone I love dying in a few decades isn't very comfortable is it?
At any rate, life, if not the best it could be, is pretty good, so regardless of what happens, I'm strapping in for a ride. I don't think there's an epoch in human history I'd rather have been born to experience really.
Well, I suppose that explains the pseudo-jazz albums about hotels on the Moon ;)
I think this is a terrible definition of a "pivotal act". When Yudkowsky suggests releasing a nanite plague that melts GPUs, he doesn't want them to melt the GPUs of the AI releasing them.
Such a decision is very much not a "one-off", in much the same way as a typical coup involves what can be roughly described as a singular act, followed by an indeterminate period of enforcement; the people who suggest it want to maintain an unshakeable technological lead over their peers, such as by making sure their AI prevents the formation or promulgation of potential peers. I don't think this is categorically bad, it depends on your priors about whether a unipolar or multipolar world is better for us, and how trustworthy the AI you're about to use is, and at the very least, if such an act succeeds, we at least have an existence proof of an aligned AGI that is likely superhuman, as it needs to be to pull that off, regardless of whether or not even better AI can be aligned. Let's hope we don't need to find out.
You are probably one of the few people who decreasd p(doom) from 2021 and after ChatGPT revolution in 2022. I updated the probability upwards due to:
The increase in capability just by adding compute and scaling the old 2017 transformer architecture was surprising to many. A moderate breakthrough in hardware can move capabilities so much? We are closer than we thought in our race to AGI. It no longer seemed to be feasible to think beyond 2100, we maybe have decades and possibly even years to do what is needed to be done. Definitely bad news for alignment timewise.
The nature of LLMs is terrible as candidate for AGI. The technology is inscrutable, explainability of these models is terrible. Nobody knows why they do what they do, nobody could predict what compute is needed for qualitative jumps such as that between Chat GPT and GPT-4. This makes the models notoriously tough to align even for basic things, like hardening them against exfiltration of training data. If AI can provide answer of when was president of France born, maybe it knows what was in the email CEO of some company sent on January 1st 2019 - if such data was used to train the model. The fact that the most likely candidate for AGI is as Yudkowsky said some just some "giant matrices of trillions inscrutable floating-point numbers" is terrifying - there may be googleplex combinations of viable matrices like that and we do not know what subset of those can be considered aligned and we do not know how to get there. We are just adding compute and are amazed that the thing that is growing in our petri dish is getting more and more capable, we do not manage the growth in any meaningful sense. In the past we had different approaches to Machine Learning specific to domains, people reasonably thought that maybe we will have to work on specific algorithm designed to integrate these separate subdomains. But no, we found out that just throwing compute on very simple game of "predict next word in text" is enough to gain multimodality and make the output more general expanding to domains like computer generated graphics, speech recognition and other areas that were previously separate fields. Also not to just talk broadly, we now know that LLM can discern self-reported race of people from images of their bones beyond current limited capabilities of medical doctors, who can do that from few things like skull etc. Nobody knows why or how the model does it, it just can and we move on.
One last thing to the above point is the old news of top-notch AI model playing GO getting beaten by one simple trick. For years people thought that the model "knew" how to play go in normal sense, the whole community thought that they "aligned" it with the task of at least beating humans in this game with very simple ruleset. Except it was proven that the model achieved results by learning some different concepts, it probably learned a completely different "game" and winning at go for years was just a sidefect. It did not learn very primitive concept that even amateurs at the game can grasp. The "alignment" of the model with basic rules of Go was a lie. I cannot imagine how can we align many orders of magnitude more complicated LLM model who has to grasp all the rules of reality, and imagine that we get exactly what we want, that there will not be any security hole and that some random teenager does not start apocalypse by some primitive prompt or strategy even after the whole scientific community will celebrate the safety of this particular "giant matrix of trillions inscrutable floating-point numbers".
We now have the new Q-Star breakthrough in Open AI. And at least according to some speculation it seems that what it achieved is that one can use compute not to train the model, but to automate evaluation of answers to questions. Imagine it as on the fly training of the model that selects most promising answers generated by larger static model in LLM powered chain-of-thought automation. It seems that this approach can temporarily boost capabilities of the model by orders of magnitude at the expense of more compute focused on specific question/prompt. If true, this means that there is another lever where you can literally throw money on some potentially productive questions like "how to make LLM more effective" and let LLM provide answers. We may be closer to intelligence explosion than we thought last year.
All in all, I do not see p(doom) decreasing in any meaningful way, quite to the contrary.
Hmm, I would say that most of those specific concerns were already priced in for me by 2021, hence why I already had such a high p(doom) at the time.
What specific means of exfiltration are you talking about? If you mean the recent claims that getting it to endlessly repeat a string will make it "leak" training data or recent interactions with other users, in the case of ChatGPT-
A) It's cost and time prohibitive to do so.
B) It's possible that the bug is with a JSON parser, or the plausible seeming outputs are just a hallucination.
If there's another way of getting it to leak training data, I can't recall one.
I've read more commentary on Q*, and the current consensus seems to be that it's not that big of a deal. I would have to look up the specific arguments, but they came from reputable sources.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link