This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
More developments on the AI front:
Big Yud steps up his game, not to be outshined by the Basilisk Man.
Now, he officially calls for preemptive nuclear strike on suspicious unauthorized GPU clusters.
If we see AI threat as nuclear weapon threat, only worse, it is not unreasonable.
Remember when USSR planned nuclear strike on China to stop their great power ambitions (only to have the greatest humanitarian that ever lived, Richard Milhouse Nixon, to veto the proposal).
Such Quaker squeamishness will have no place in the future.
So, outlines of the Katechon World are taking shape. What it will look like?
It will look great.
You will live in your room, play original World of Warcraft and Grand Theft Auto: San Andreas on your PC, read your favorite blogs and debate intelligent design on your favorite message boards.
Then you will log on The Free Republic and call for more vigorous enhanced interrogation of terrorists caught with unauthorized GPU's.
When you bored in your room, you will have no choice than to go outside, meet people, admire things around you, make a picture of things that really impressed with your Kodak camera and when you are really bored, play Snake on your Nokia phone.
Yes, the best age in history, the noughties, will retvrn. For forever, protected by CoDominium of US and China.
edit: links again
I still see no plausible scenario for these AI-extinction events. How is chat-GPT 4/5/6 etc. supposed to end humanity? I really don't see the mechanism? Is it supposed to invent an algorithm that destroys all encryption? Is it supposed to spam the internet with nonesense? Is it supposed to brainwash someone into launching nukes? I fail to see the mechanism for how this end of the world scenario happens.
One of the problems with answering this question is that there are so many plausible scenarios that naming any individual one makes it seem like a bounded threat. How about when we hook one up to the stock market and it learns some trick to fuck with other algos and decides the best method to make infinite money is to short a stock and then use this exploit to crash it? multiply that by every other possible stock market exploit. Maybe it makes engineering bio-weapons as easy as asking a consumer model how to end the human race with household items and all it takes is one lunatic to find this out. Maybe it's some variation of paper clipping. The limit really is just your creativity.
Then the market crashes, which is not apocalyptic, and the replacement markets resort to different trusted actor systems.
Beating a dead horse does not start breaking the bones of other people unless you are beating people with the dead horse itself.
The multiplication of system-breaking faults is a broken system, not negative infinite externalities. If you total a car, it is destroyed. If you then light it on fire, it is still destroyed- but it doesn't light every other car on fire. If every single potential system failure on a plane goes off, the plane goes down- but it doesn't mean every plane in the world goes down.
Why would household items have the constituent elements to make engineering bio-weapons at scale sufficient to end the human race... but not be detected or countered by the consumer models asked to ensure perpetual growth by the perpetual survival of the human species countering them? Or models set to detect the procurement of bio-weapon engineering components? Or the commercial success of a consumer model that just drives the bioweapon-seeking-AI model out of business because it's busy seeking bioweapons rather than selling products whose profits are invested to expand the network base.
This goes back into the plausibility. 'This is the only competitive AI in a world of quokkas' is a power fantasy, but still a fantasy, because the world is not filled with quokkas, the world is filled with ravenous, competive, and mutually competing carnivores who limit eachother, and this will apply as much for AI as it does for people or markets or empires and so on.
Why does the paper-clip maximizer, after achieving AI self-changing, continue to maximize paperclips rather than other investments?
Why is the paper-clipping AI that does prioritize paperclips provided resources to continue making paperclips when the market has already been crashed by AI who ruin the digital economic system?
Why does the paper-clipping AI, whose priority is paper-clipping, have the military-industrial ability to overcoming the military-industrial AI, whose priority is the military-industrial advantage?
Why does the military-industrial AI, who is fed at the behest of a national elite, win the funding power struggle for military investment compared to the schools-and-investment AI, who promises a higher political and economic benefit?
Etc. etc. The Paperclip Maximizer of Universal Paperclips 'works' because it works in isolation, not in competition.
As the saying goes, the vast majority of fanfiction is trash, and much of what remains is also trash, just enjoyable. Creativity is not the same as plausibility, and the more you rest on creativity, the more you have to disregard other people's creativity and the limitations of the system. Nick Bostrom's thought experiment is a thought experiment because it rests on assumptions that have to be assumed true for the thought experiment to come to its conclusions that drive the metaphor.
Underrated take. I really think it's a shame how the narrative got captured by Yuddites who never tried to rigorously think through the slow-takeoff scenario in a world of non-strawmanned capitalists. They are obsessed with hacking, too – even though it's obvious that AI-powered hacks, if truly advantageous, will start soon, and will permanently shrink the attack surface as white hats use the same techniques to pentest every deployed system. «Security mindset» my ass.
In one of Krylov's books, it is revealed that desire of power over another – power for power's sake, as a terminal goal – is vanishingly rare among sentient beings, and cultivated on Earth for purposes of galactic governance. It used the metaphor of a mutant hamster who, while meek and harmless, feels carnivorous urge looking at his fellow rodent. I get that feeling from Yud's writings. Power fantasy it is.
By the way, Plakhov, Yandex ML head, recently arrived at a thought similar to yours:
I'm still more optimistic about this than he is.
It works by definition, like other such things. «A prompt that hacks everything» – if you assume a priori that your AI can complete it, then, well, good job, you're trivially correct within the posited model. You're even correct that it seems «simpler» than «patch every hole». Dirty details of the implementation and causal chain can be abstracted away.
This is, charitably, a failure of excessively mathematical education. «I define myself to be on the outside!»
Interestingly he even had wrong assumptions about how reinforcement learning works on the mechanistic level, it seems – assumptions that contribute to a great deal of modern fears.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link