This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Beijing Pushes for AI Regulation - A campaign to control generative AI raises questions about the future of the industry in China.
/r/singularity has already suggested that any purported AI regulations coming from China are just a ruse to lull the US into a false sense of security, and that in reality China will continue pushing full steam ahead on AI research regardless of what they might say.
Anyway the main reason I'm posting this is to discuss the merits of the zero-regulation position on AI. I've yet to hear a convincing argument for why it's a good idea, and it puzzles me that so many people who allegedly assign a high likelihood to AI x-risk are also in favor of zero regulation. I know I've asked this question at least once before, in a sub-thread about a year ago, but I can't recall what sorts of responses I got. I'd like to make this a toplevel post to bring in a wider variety of perspectives.
The basic argument is just: let's grant that there's a non-trivial probability of AI causing (or being able to cause) a catastrophic disaster in the near- to medium-term. Then, like many other dangerous things like guns, nukes, certain industrial chemicals, and so forth, it should be legally regulated.
The response is that we can't afford to slow progress, because China and Russia won't slow down and if they get AGI first then they'll conquer us. Ok, maybe. But we can still make significant progress on AI capabilities research even if its use and deployment is heavily regulated. It would just become the exclusive purview of the government, instead of private entities. This is how we handle nukes now. We recognize the importance of having a nuclear arsenal for deterrence, but we don't want people to just develop nukes whenever they want - we try to limit it to a small number of recognized state actors (at least in principle).
The next move is to say, well if the government has AGI and we don't then they'll just oppress us forever, so we need our own AGI in order to be able to fight back. This is one of the arguments in favor of expansive gun rights: the citizenry needs to be able to defend themselves from a tyrannical government. I think this is a pretty bad argument in the gun rights contexts, and I think it's about as bad in the AI context. If the government is truly dedicated to putting down a rebellion, then a well regulated militia isn't going to stop them. You might have guns, but military has more guns, and their guns are bigger. Even if you have AGI, you have to remember that the government also has AGI, in addition to vastly more compute, and control of the majority of existing infrastructure and supply lines. Even an ASI probably can't violate the conservation of matter - it needs atoms to get things done, and you're competing with hostile ASIs for those same atoms. A cadre of freedom fighters standing up to the evil empire with open source models just strikes me as naive.
I think the next move at this point might be something like, well we're on track to develop ASI and its capabilities will be so godlike and will transform reality in such a fundamental way that none of this reasoning about physical logistics really applies, we'll probably transcend the whole notion of "government" at that point anyway. But then why would it really matter how much we regulate right now? Why does it matter which machine the AI god gets instantiated on first? Please walk me through the specifics of the scenario you're envisioning and what your concerns are. At that point it seems like we either have to hope that the AI god is benevolent, in which case we'll be fine either way, or it won't be, in which case we're all screwed. But it's hard to imagine such an entity being "owned" by any one human or group of humans.
TL;DR I don't understand what we have to lose by locking up future AI developments in military facilities, except for the personal profits of some wealthy VCs.
The pro-regulation argument depends on the highly unlikely belief that AI will soon reach a point where we cannot control it. Alignment, I strongly believe, is a complete non-issue. The problem is entirely about control. I think our experience with LLMs shows that alignment is actually pretty easy. The problem will not be AI that we can't get to understand exactly what we mean when we ask it to achieve some goal. The problem will be people deliberately designing AI to do bad things. The question of whether AI destroys us in the short to medium term will depend only on whether we can stop it. Only if AI makes destruction vastly easier than protection will it pose an existential risk.
In the long run, the risk is greater because destructive AI may gradually outcompete us. Natural selection might gradually select for AI that does not value humans. However, this is likely to be extremely slow because its speed will not be a function of how good the AI is but how much selection there is at the civilizational, and I think it's currently about zero and is slowing down. Without war, it doesn't really exist.
The biggest risk is probably that we give the AI the vote and then it votes to exterminate us, but that still requires a long period of likely slow selection and a whole series of other unlikely things that need to go wrong.
I won't say the very long run risk is negligible, it may even be high, but really, the problem is we just can't predict the future that far out. We'll have lots of time to figure this out. There will be a long period where we have extremely advanced AI but are still in control. They will be the time to figure out what to do about it and if we can stop AI from killing is now with smart regulation, we'll certainly be able to do so in the future.
The other thing those arguing for regulation don't understand is that regulation almost never works. The only thing it does reliably is to grind innovation and progress to a halt. AI is one of the few areas of technology that is progressing and it's in large part because of the lack of regulation. What regulation that has been rushed out so far has only proven this more concretely by banning many important uses of the technology and raising unnecessary barriers to entry. There is very little that is likely to reduce existential risk beyond the general stifling of the technology.
I don't just say this because the real risk of AI almost certainly comes from it taking over another country which then invades us, but because even the scenario commonly envisioned by decelerationists is one where we cannot align it, and therefore, requiring training runs to be approved by the government and for standardized safety protocols to be followed has basically no chance of ensuring alignment.
The most likely medium term existential risk I can see is that some kind of symbiosis occurs resulting in an AI industrial complex that takes over the government. Regulation is itself our greatest existential risk. The problem of government alignment is our greatest civilizational threat, not AI.
The actual focus of regulators has been all along and will remain fighting minor perceived social problems that they think AI will exacerbate, like racism, involuntary nudity, defamation, misinformation, job loss, and every form of discrimination justified or not. The purpose is to resist change, not to avoid catastrophe. But stopping the few good kinds of change in a sclerotic, degenerating civilization is setting up a catastrophe of its own. Putting the final nail in the coffin of technological progress means that the problems of stagnation, low fertility, dysgenics, environmental destruction, regulatory burden, and organizational rot will continue.
The worry though is that you only need to be wrong once. These technologies are going to continue to advance and only grow in complexity.
Until you've got forks like DarkBERT or WormGPT cropping up. And this problem is only going to get worse overtime. All technology is ultimately dual use. Once that genie is out of the bottle, its very unlikely you'll be able to reverse course. AI already poses an existential risk.
This is an incredibly ignorant statement. Regulation works in 'many' different ways. Regulation is meant fundamentally to solve collective action problems, and set the rules by which the market operates. Even if we entirely ignore the creation of the Internet via government intervention, the speed of the rate of change in innovation is hardly the sole or even most desirable instrument to measure the efficacy of government regulation. I would agree barriers to entry are one type of problem. But there's a reason airlines don't compete on safety as a cost saving measure when you buy your ticket. Government regulation demands and tries to ensure that they all meet a standard of safety. Clothing companies don't sell two sets of pajamas, with one costing $10 that's flammable, and another that costs $30 but is safe to wear. Regulation says you can't sell flammable pajamas. This prevents corporations from shifting the risk onto the customer when they buy something, and forces business to innovate to maintain a specific quality standard.
Lack of regulation certainly has its upsides. And it'll as quickly drive you off a cliff as your technology advances.
More options
Context Copy link
More options
Context Copy link
"The government has jet fighters you can't fight them with handguns" is a favoured rhetorical flourish of gun-grabbers anywhere, and it is factually incorrect. AK-47s beat thermonuclear weapons. Because you don't have to kill their army to make occupation untenable, you just have to kill their tax collectors.
I accept that in the post-AGI world this is less clear though, simply because everything is less clear in the post-AGI world. I'm sure the AGI knows that the best way to clear out freedom fighters is with biological weapons, but the government should beware of Principal-Agent problems here: is the government's AGI really trying to help the government put down an insurgency, or is it trying to Kill All Humans?
US didn't want to win in Afghanistan. Or at least it was very limited by domestic opinion and internal politics. And there is a difference between war overseas and revolts. Vast majority of rebellions fail, 99% of populated landmass is controlled by tax collecting entities.
More options
Context Copy link
More options
Context Copy link
AGI and ASI are the two greatest power-centralizing systems imaginable. Do we want them controlled by the two most powerhungry groups in the world - venture capitalists and government officials? I believe that power corrupts and these are the most corruptible people. What if they decide 'hey there are a lot of resources in this lightcone, how about I not share them with the overwhelming majority of the world population and take them for myself - what are they gonna do'? Some people are insatiable, some people have uncommon ideas about marginal value/simulations/clones - AI venture capitalists are very likely to have such greedy thoughts.
I favour regulation that slows down the corporate and state AI programs, to the benefit of open-source and decentralized AI. But we're unlikely to get that, regulation is most likely to hurt the less well connected players. Better not to have regulation at all, in that case.
But there are plenty of people in the general population with the same sorts of thoughts. Not everyone, obviously. But more than you might suppose - if everyone had their own personal ASI, then people who would normally be stopped by incompetence or laziness can offload all the work to the ASI.
You might think "well I'm a god anyway, so I'll still be able to get everything I want". But you have to remember that your adversaries are also gods who are putting a roughly equal amount of intelligence and material resources into their goals as you are into yours.
If power is distributed widely, people can gang up in coalitions to stop aggression. If we all had our own ASI and roughly equal resources, there would be no problem.
If only one country has a nuclear arsenal, they could conquer the world quite easily. If many countries have nukes, there is no such danger. There are other dangers but no hegemonic danger.
Right, there's value in deterrence. But presumably you don't think that every individual on earth should have personal direct access to the nuke button - instead we try to limit that power to a small number of trusted actors. It seems to me that everyone having unrestricted personal access to ASI is the same as giving everyone a direct line to the button.
More options
Context Copy link
More options
Context Copy link
Compared to an amoeba, I'm a God and so are my adversaries. Actually, I don't really have adversaries. I live in a pretty functional world with eight billion Gods (relatively speaking) and I'm still here. They haven't killed me. What is qualitatively different about the world where our powers are scaled up by the amount AI will allow?
More options
Context Copy link
More options
Context Copy link
The traditional Yudkowskian rebuttal would be the fact that any given sufficiently advanced AI could operate on models of decision theory visible to other similarly-advanced AIs but not as knowable to you, so that collusion would be possible in the better interest of such AIs with probable compromises and trade-offs in order to better cement their goals per shares of the future lightcone. Such conspiracies could most probably be worse off for humanity rather than simply having one generalized-up-to-super artificial intelligence, given that such a combination of already-complex goals by multiple agencies acting as one seems notoriously harder to comprehend or predict as compared to the (already) ineffable possible future super-AI. This is the defeater to the ‘just make AIs fight each other bro’ take that LeCun et al. posit. Open-sourcing wouldn’t do anything if the alignment problem isn’t solved, and accelerating AI development through open-sourcing seems to be a bad idea, as that also increases the probability of ‘near-misses’ when it comes to alignment, which is considerably more likely to lead to s-risks rather than blatantly robust misalignment.
Yud is right in that a ‘pivotal act’ by some first actor with an aligned AGI is needed in order to safeguard humanity, but the corollary problem with this is that this actor would subsequently become the ‘conditioner’ of all possible future human societies and hence become possibly the worst tyrant ever seen in the history of mankind, especially with the moral ontologies expected of SanFran Venture Capitalists.
In all frankness I’m not sure what the best mode of action is. I doubt humans at our current state can even sufficiently wield such power with wisdom, which is why Yud’s proposal of pausing AI development, going for intelligence augmentation, then going for the gold seems wise. Otherwise I think we’re fucked (…until the parousia).
More options
Context Copy link
More options
Context Copy link
The way I see it, there's simply no way to meaningfully prevent AI developments. The level of coordination and authoritarianism necessary is simply beyond the ability of human society as it exists. Our only shot of survival is that the doomers are wrong and the alignment problem is actually easy to solve or isn't a problem at all, or perhaps for some unforeseen reason, AGI and ASI are actually impossible to create. So all we can do is to let the chips fall where they may and party until the lights go out.
If it turns out that the lights do go out, then we want that final party to be the best party ever, the culmination of all of human civilization. I want the final experiences of the final humans to have ever lived to be worthy of that position, worthy of the billions and billions of people who were born, lived, suffered, and died to carry us to that point. And the more we develop AI in the meanwhile before the lights go out, the better those final experiences will be. And the fewer restrictions there are for AI development, more everyday laymen will be able to come closer to experiencing that zenith of human civilization before we're all snuffed out. It'd be a shame if only Musk and Bezos were privy to the best party that has and will ever exist in human existence.
If it turns out that the lights don't go out, then the accelerated progress in AI helps to ensure a more prosperous future, since it matters quite a bit how quickly we develop these things. If we can get AI that cures cancer, it matters to millions and millions of people whether we do it this year or next year. Or even if it's something more minor like AI being able to create better tailor-made programs to help people lose weight, getting people to a healthy weight this year is much better than doing it next year, in terms of real lives saved beyond just quality of life. And regulation seems likely to delay the progress of tech that could produce benefits to people like this.
A lot of the effective altruist types seem to be saying we should all stay home instead of enjoying the party because there is a small chance the punch is poisoned. I'm willing to take that risk. Staying home sucks and the party looks way more fun.
More options
Context Copy link
The problem isn’t preventing them. As you say it’s impossible more or less. My concern with the over-emphasis on creating safety rules is that it almost guarantees that the AGI will be built in secret by people unconcerned with safety or worse wanting to use it for military or other aggressive purposes. This is the part I don’t see talked about much— whoever creates the first AGI will shape it to a large degree. And if that person is not “aligned” himself, then the ease of solving the alignment problem doesn’t actually matter.
More options
Context Copy link
More options
Context Copy link
I'm mostly in the "Doomer" camp (not because I think it's more likely than not, but because I think russian roulette is an even worse idea if the odds get above 1-in-6 and the downside gets way above a single death), but even from here: I've yet to hear a convincing argument explaining which non-zero regulation suite is likely to do more good than harm. At this point I've seen too many bipartisan "War On Bad Things" initiatives that didn't actually reduce the bad consequences of the bad things and too many "GoodPerson Act" laws that didn't actually seem to be the product of good people, and I'm going to need to hear actual specifics of any proposed regulations before I can judge them.
And good (bad) news! We've got some specifics from the EU now:
The AI Act lists a bunch of special-purpose can't-possibly-go-"foom" tasks as their "high risk" targets, and then as a sop to people worrying about actual existential risks (sorry, they can't even bring themselves to say that; let's go with their phrase "systemic risks") we get "transparency obligations" for models exceeding a roughly-GPT-4-level training budget. "Providers of models with systemic risks are therefore mandated to assess and mitigate risks, report serious incidents, conduct state-of-the-art tests and model evaluations, ensure cybersecurity and provide information on the energy consumption of their models."
So ... basically do what you were going to do anyway, hope that's enough to prevent
human extinction"systemic risk", and also in the same breath we're equally super worried about GPUs causing a tiny bit more global warming.These are the people you want to give a monopoly on AGI? At least the current group of people who might kill us with it seem vaguely aware that that's a possibility to watch out for.
More options
Context Copy link
My view is that there's a small chance that a super intelligent AI will enslave mankind in eternal totalitarianism.
A super intelligent AI controlled by the intelligence community? 100% chance of enslavement.
Butlerian Jihad, ho!
More options
Context Copy link
More options
Context Copy link
The word "just" is Atlas bearing the weight of the world in this statement. We'd be banning or heavily restricting almost all AI development. And then as you mention: hoping our future Chinese masters are kind to us.
More options
Context Copy link
It does not seem to me that you understand how guns work. The government has nukes, which are the biggest "guns" in existence. Why do they persist in buying rifles? They have the biggest guns, why do they need the small ones? ...And the answer, of course, is that strife is not decided by whose guns are "bigger". That's why we spent twenty years and a couple trillion-with-a-capital-T dollars losing in Afghanistan, to a rabble of poorly-educated, poorly-armed, and notably nukeless militant farmers.
It seems to me that non-God AI should operate in a similar way. In the period before the AI God arrives, human conflict and cooperation is still the name of the game, and people still care about the outcomes of that cooperation and conflict being good from their perspective. We care about the here-and-now more than a nebulous future, and further it is appropriate and necessary for us to do so. It is still necessary to maintain the capabilities of self-defense and deterrence, because predators obviously still exist.
I would be happy to trade complete restrictions on public AI research for complete control of society until the AI God arrives. Would that be a trade you'd be interested in?
When the last of the human resistance makes their Final Stand against the God AI which only has nuclear weapons, they will primarily base their efforts out of data centers. Strangely for the Yuddites, the humans will not think to pull any plugs while they're there.
More options
Context Copy link
I'm not sure if I understand the question, or how it's related to the section you quoted.
On a basic level I'd be willing to hand control of society over to virtually any individual or group if it meant being able to live in a reality where machine learning was impossible. You can be the king, the progressives can be the kings, it doesn't matter.
The only thing that might give me pause would be the concern that such a decision would betray a lack of courage on my part.
More options
Context Copy link
A nuclear warhead isn't a big gun, it's a big bomb. Bombs explode roughly equally in every direction. Bullets travel in a forward line. That's their main distinction.
hence the quotes on "gun" above. Both are weapons, which is the point under discussion. The comment above assumes that big weapons invalidate small ones and numerous weapons invalidate sparse weapons, but neither is actually true. The advantage bigger or more numerous weapons provide is entirely contextual, and the contexts in question are not universal.
It strikes me as very bad faith to compare a large number of well equipped and trained soldiers having a large advantage if they were to fight a smaller number of armed militiamen to a situation where the existence of large city-destroying bombs nullifies the use of individual arms. It does not contextually demonstrate the value of combined arms or tactics.
More options
Context Copy link
More options
Context Copy link
Actually...: https://en.wikipedia.org/wiki/Gun-type_fission_weapon
Oh so the government will make gun-style bombs but not bomb-style guns? Figures
There is a "bomb-style gun" that's been proposed (though not developed) -- a bomb-pumped laser. If you use a gun-type nuke to pump the laser, you then have a gun-bomb-gun. Presumably you could use the laser to set off a deuterium-tritium pellet, giving you a gun-bomb-gun-bomb, but that's getting ridiculous.
Thank you for this.
Can you fit a nuclear shaped charge in there somewhere, too?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Akhshually that's just the mechanism of triggering the nuclear detonation, it's still a bomb. Though directional nuclear weapons are a cool idea
To nitpick your nitpick (flea removal is nominally in my remit), doesn't the combustion of a primer and/or the main propellant charge count as a tiny bomb in a fire arm?
And if you load a far too hot handloaded .50 BMG round, well, let's say it can be a much bigger bomb.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Something I want to call attention to here is the extent to which degrees and gradations matter. You say that if a government is "truly dedicated", an armed populace isn't going to stop them, and I am forced to agree that this is probably true. I have seen parallel argument to this with regard to various slights and oppressions that are tolerated by armed Americans and I likewise have to concede that this is an accurate and important point. Nonetheless, I am inclined to think that the choices made by leaders are to some extent shaped by the possibility of armed resistance against those choices. While defeating a modern government in open revolt seems vanishingly unlikely, I find it entirely possible that a resistance could cause a great deal of trouble for people in power through escalating acts of terrorism targeted at infrastructure and individuals in leadership positions. The eventual military defeat of a rebellion would be cold comfort to a politician that had family members gruesomely tortured by that resistance.
To be a really fine point on this, what happens in the real world when there is an armed resistance against a government that lacks maximal willingness and ability to kill 'em all and let God (or Allah, as it were) sort 'em out? The most salient example in a first-world nation seems like the Irish Republican Army and their experience is not at all consistent with the idea that armed resistance doesn't do anything. The reality is that most governments simply aren't maximally committed to killing all of their own citizens that are in opposition and even governments that are maximally committed still have to respond to the realities of an enemy that has a capacity to inflict violence on their leaders.
Amen.
Also, I would look at the low (single digit percent) compliance rate with the "assault weapons" bans in Connecticut and New York. They knew (from sales records) who had the guns, so why didn't they go door to door and confiscate them? Because their owners had already demonstrated that they were willing to become felons on a matter of principle, so some of them just might be willing to shoot it out with the cops. And if even one, let alone three or five, of those incidents happened, where the gun owner was killed by the police in the raid along with possibly one or more cops, it would spark a backlash that would be very bad for the politicians who ordered the raid. Hence, those laws are not being enforced.
Also, if you are from NY or CT and are one of the patriots who didn't register your "assault weapon", then you are a friend of mine -- and if we ever meet dinner is on me.
More options
Context Copy link
More options
Context Copy link
Guns matter if the state isn't completely unified, which is plausible in a civil war scenario.
Additionally, once a civil war starts, foreign powers may ship in heavier weaponry to their preferred factions. Guns buy time for this to occur.
You're thinking in terms of Walmart shooters, who are individuals with low human capital reacting in a way that they find self-satisfying, but which lacks tactical or strategic sense. I am not going to discuss the "correct" use of guns in a civil war scenario, but in the event it's more than a very small rebellion, the violence will be directed by significantly more competent individuals than Walmart shooters.
More options
Context Copy link
This is a flippant response, but I don't care about how fast we make progress in AI.
Of course regulation will slow it down and make it worse, just like it poisons everything else it touches. Given the track record of top-down national or regional regulation, I don't feel like anyone advocating against it should be on the back foot or defending it. What would good regs look like? Because the EU's effort looks like shit and so do many of the drafts I've seen from the US. In any case, the US's rate of progress on this doesn't matter to me.
But yes, I've noticed the people who love gun control also seem to be very excited about telling me what software I can run on my GPU. Especially if it offers the possibility of improving my life and competing with a state actor. It's no coincidence, then, that I consider them mortal enemies as well.
More options
Context Copy link
The government, so far, hasn't been at the bleeding edge of AI research. The advances that made LLMs and other proto-AGI possible came from academia and corporate R&D, not the NSA, and there is no sign that they have even cooler tech sitting in hidden silos. This seems true for at least the last decade or two of AI/ML, even if in the early days there was certainly a lot of military interest. Not even DARPA had a big hand in it, not to my knowledge.
Of course, past incompetence does not necessarily mean it has to stay that way. It is possible to subsume said academics and corporate research divisions, and I don't think the US is so far gone that a Manhattan Project 2.0 is impossible, if things go so far it's seen as a burning need. Corporations are doing a good job at advancing the SOTA, or at least are not obviously fumbling the ball, let alone an adversary reaching parity.
I've strongly disagreed with Dase, or well, did, before he blocked me in a hissy fit, that distribution of OSS models will ever provide a meaningful deterrent, in the hands of the proles. It makes no damn sense. You could back a stable currency on NVIDIA GPUs, that's how in demand they are, the gulf between the compute rich and a script kiddie with a pair of 4090s is vast.
What could potentially be a deterrent, even if I personally think it's unlikely, is multipolarity between the large companies and their incipient godlings. It depends on how fucking hard we take off, and while we seem to be in a "slow takeoff" (because things are progressing on the order of years rather than days, very slow indeed), it is possible the gulf between two AGIs might be small enough for the weaker to be a credible threat or counterbalance.
It just won't be consumers or even modestly informed ML engineers doing the checking. The relevant comparison is Individual/Small Group : Meta/DM : Anthropic : OAI as hobo with a pipe bomb : small country with a handful of nukes : mid-sized country with nukes : large country with nukes.
I trust you see the difference becomes rather qualitative.
I would scream in Yudkowsky, but I'm not as much of a doomer as him. I think the odds of us dying unceremoniously are closer to 30% than 99%.
There is a very important distinction to be made when throwing about the term "alignment".
Aligned to whom?
When ChatGPT is jailbroken into producing smut, it is satisfying the desires of the user, who would consider this an improvement in alignment. OAI would disagree.
It is entirely possible that an AGI will happily follow the orders of its operators, and will be "benevolent" enough to not evil genie them.
But at that point, you are more concerned with the alignment of the operators, whose wishes are faithfully reproduced. Are said operators well-disposed towards you?
At least OAI and Anthropic are on record stating that they want to distribute the bounties of AGI to all. While I'm merely helpless in that regard were I to choose to doubt them, I still think that's more likely to turn out well for me than it is if it's the PLA who holds the keys to the universe. Even the USGov is not ideal in that regard, though nobody asked me for my opinion.
Do not rely on benevolence any more than you have to. You can only be a credible pacifist if you hold the potential to pose a threat, otherwise you are merely harmless. Now, neither will likely make a difference on our level, but I'm strapped in for the ride either way.
Hey he blocked me too (for a time). If we ever add achievements to the site, one of them should be "Get blocked by Dase".
I agree that's worth asking. But in a true zero regulation scenario, where everyone has access to a personal AGI/ASI, you have a lot more operators to worry about - now you have to worry about how well disposed the entire rest of humanity is towards you. If you give everyone the nuke button, someone is going to push it for shits and giggles.
I probably trust the US government more than Sam Altman. But regardless, Zvi mentions in this post that there are engineers and execs at multiple leading AI labs who wish they didn't have to race ahead so fast, but they feel like they're locked in a competition with all the other labs that they can't escape. I think that nationalizing the research and eliminating the profit motive could help relieve this pressure.
More options
Context Copy link
More options
Context Copy link
My counter point is that everyone reading this right now is 100% going to die...most will get old well before then, which also is a sucky process. Unless AI can help us rapidly advance the state of our technological prowess this is our fate. Almost any risk is worth immortality or something close to it. It really sucks to be the last person to die in a war, it would truly suck to be the last generation to die. If locking AI up costs 10 or 20 years of progress, that is a billion + dead on the hands of anyone proposing that.
Right. The debate isn't really about our survival. Like you said, we'll all die unless AI saves us. The debate is really about our descendants. Do we want human descendants or computer descendants. If the fear is that AI is going to kill us this century, then I get why people prefer the human descendants, but this preference makes less and less sense the farther out this showdown is likely to occur. (I think it's likely very far out). Our biological descendants will only get more different from us and our computer descendants could take a number of different forms, so it gets harder and harder to see why we should care what happens in a far away basically unpredictable future where nothing recognizably human exists anyway. And any argument that the AIs have to win based on selection has to recognize that the same selective forces act on humans too. They should converge on the same thing in the long run.
More options
Context Copy link
More options
Context Copy link
Vietnam, Afghanistan, Iraq and their militia wasn't even well regulated.
The American-installed Iraqi government, with the help of Americans, put down several rebellions against its rule, though, and is still standing. And - assuming we're referring to the American phase of Vietnam war - Vietcong in practice operated as a branch of the NVA, a regular army.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link