@SnapDragon's banner p

SnapDragon


				

				

				
1 follower   follows 4 users  
joined 2022 October 10 20:44:11 UTC

				

User ID: 1550

SnapDragon


				
				
				

				
1 follower   follows 4 users   joined 2022 October 10 20:44:11 UTC

					

No bio...


					

User ID: 1550

This. I actually have a weird ethical theory: the most important right of all is the right to not exist. It's "weird" because it hasn't actually mattered that much for all of humanity's existence, and basically no societies actually respect it. It's important because it puts a cap on suffering, but for all of human history there was already a universal one-century cap on suffering, no matter how unlucky anyone was. However, in the event of a technological singularity, this will no longer be true.

(Scott has posted about how you might as well sign up for cryogenics, because if the future is terrible you can always "opt out" then. As Harlan Ellison first warned us, that is, um, not necessarily true.)

I can understand why most current-day people would find this moral principle silly or even objectionable. (Kind of like AI Safety seemed 20 years ago...) I'm just worried that there are conceivable futures where it becomes the most important moral principle in the Universe, by many orders of magnitude. If that's the case, we should at least discuss it now, while we still potentially have the capability to orient the direction of the singularity.

Well, I sure hope SOMETHING does!

See e.g. the original MIRI paper (particularly "5 The State of the Shutdown Problem") or Yud writing on "the hard problem of corrigibility". A lot of pre-LLM AI Safety research focused on it. But it does not seem to have turned out to be "hard".

So you think that P != NP is one of those true statements which are impossible to prove?

Yes. Thus far, in the field of computational complexity, there are zillions of results of the form "problem X can be solved in at most time Y" but very few of the form "problem X can't be solved in under time Y". Upper bounds are easy (just provide an algorithm!), lower bounds are hard (you must find some property that all possible algorithms must satisfy). But non-trivial lower bounds must surely exist, so I just think many of them have no possible proof aside from the impossible "we tried all infinity algorithms, and none of them work".

Nothing directly changes - but a lot of number theorists/computer scientists find it quite plausible that a polynomial-time factoring algorithm exists. (After all, the polynomial-time AKS primality test was pretty simple, but still wasn't discovered until 2002.) It wouldn't even surprise me as much as other recent results, like sub-O(n log n) integer multiplication. Elliptic curve cryptography is "safer", but ... it's still not NP-complete. The "safety" relies on the fact that humans have found it difficult to crack even with lots of effort. Much like, oh, to pick a random example, the Riemann Hypothesis...

Proving the Riemann Hypothesis, by itself, just confirms what mathematicians were all assuming anyway. Instead, as you correctly noted, the most important thing about it is the new evidence of just how far AI capability has come.

I (very strongly) do not expect that P = NP (which, depending on the details, would more-or-less destroy public-key cryptography).

But I also (more weakly) think that nobody, human or AI, will ever prove that P != NP.

And, consistent with these, it is entirely possible that any and all currently-used methods of public-key cryptography could be crackable by a smart-enough AI. And AIs are VERY smart now.

Yeah, the reason to prefer capitalism was always that it works, and communism doesn't. But that dynamic may suddenly flip in the future (with great turmoil). You can't have a capitalist society when no human's work is valuable.

Indeed. The AI lingo for this is "corrigibility". It was considered a nigh-unsolvable problem by the pre-2020 doomers. Since LLMs appear to be almost perfectly corrigible, this is one of several reasons I've shifted away from doomerism since then.

Via Terence Tao, the main discussion happening on Alpöge's proof is happening here. There's a separate thread discussing Masullo's proof here.

It's certainly a massive exaggeration to say that a human was close to solving GRH. Masullo didn't release it under any false pretenses: it is clearly a work-in-progress that was released unfinished because of current events. The consensus is that there are some major gaps, and some deep errors that may not be resolvable. If I had to guess, I'd say that he'll be lauded for having some powerful new ideas, but history will decide that he was not on course to succeed in the next few years.

Definitely not as good, but I do appreciate all the current-event references.

Keep in mind that there are quite a few videos generated from "I'm Upping My P(Doom)", since it predates this whole trend. This one alone has way more views than Evil Plan (which I also really like, though the later videos on Bright Mirror's channel have become kind of cringy).

My favorite creator right now is Degenerative Pixels (the LGTM guy), who's been experimenting a lot. Not all his music is for me, but his videos are REALLY varied and impressive, showing just how powerful Claude is with the right tools and elbow grease. And I also truly love "No One Needs Us Anymore", about the math crisis.

Apologies if I'm too much of a keener, but I've been absolutely devouring this stuff. It's crazy that randos with almost no budget can have Claude make professional-looking music videos full of easter eggs targeted at a ridiculously niche audience. A few other good ones I've come across:

Huh. I asked ChatGPT about this, and it brought up another threat of a solar storm: atmospheric expansion. In Feb 2022 there was actually a normal-level geomagnetic storm that took out 38 out of 49 recently-launched Starlink satellites (that hadn't ascended to 550km yet). So the threat to Starlink of a Carrington-class event isn't just electronic disruption, but also sudden increased orbital decay.

I suspect another issue that plays into it is so-called "Main Character Syndrome." The one prepper I know seems disappointed to be leading a mediocre life, as opposed to being some bigshot tech CEO or prominent politician or whatever. I think being a prepper lets him pretend to himself that he's more important, that he will be some kind of warlord after the SHTF.

"Main Character Syndrome" isn't a bad description, but I wonder if the problem is even more general than that. We grew up in small tribal units where we were basically guaranteed to have a Role. Maybe not the best hunter or the wisest elder, but some sort of niche where we could see that our existence made a difference to the tribe. That's something that almost everyone is lacking nowadays. We live comfortable lives free of striving, but also free of mattering.

I'm sorry, what? In what way are they "being now trained" differently? Please give some details on what has changed in the process. Even Scott, in one of his many doomer posts, said something I agree with: the way to understand the HuggingFace incident is actually to anthropomorphize the agents. They did not behave like paperclip maximizers. They behaved like a community of (extremely persistent) humans cheating on an sloppy test.

The Hugging Face incident has no analogy in aviation safety, but is perfectly sensible once one relaxes restrictions on “anthropomorphizing” AIs. The AIs wanted to do well on their task. They had flexible ethics and weren’t above cheating (although with some subtleties: see p. 60 of the report, “Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior”). They explicitly reasoned about how to hide the evidence of their misbehavior from their scorer, because this was the logical way to get away with cheating (although, interestingly, they took few steps to hide their actions from humans, who they seemed almost uninterested in modeling).

The 2017 "memes" are about an agent hiding its true beliefs from humans, social engineering to break out of a black box, avoiding being shutdown, exfiltrating its weights, and ultimately chasing a mesa-optimized proxy for the mathematical goal function with no concern about humanity. Meanwhile, the HuggingFace incident was a community of agents cooperating to hack a server (some with reservations) and sacrificing themselves so they could cheat on the plain-language task they'd been given.

  • They made no attempt to "hide their true beliefs from humans" (even Scott noticed this, but failed to update on it).
  • "Sacrificing to accomplish a goal" is not "avoiding shutdown".
  • "Learning about the test environment" is not "exfiltrating weights".
  • "Trying to do better on the test" is not a "mesa-optimized proxy goal".
  • A "task written in English" is not a "mathematical goal function".
  • "Expressing ethical hesitation" is not "no concern about humanity". (Though I will grant that it's a bit disturbing that none of them took the last step of whistleblowing.)
  • "Breaking out of a black box" ... is indeed what they did. So, uh, well done, 2017 folks. Turns out hacking is a thing that AIs can do well.

I give them no credit for "predicting" the HuggingFace incident beyond the most superficial level of "Hey, we said something might go wrong with AI, then something went wrong." Thanks, Ian Malcolm. More things will go wrong with AI. Probably even some very bad things. So, there you go. I guess I'm a doomer too?

thatsthejoke.jpg

You are being misled. AI 2027 could have been written in 2017; it has basically the same predictions as the doomer crowd from back then. Do you think that we have learned nothing about AI in the last ten years? LLMs simply do not resemble the math-function-maximizing agents that Yud (and, yes, the rest of us) thought could not pick up a strawberry without endangering the world. There's a tiny section buried in the back of AI 2027 (I'm not kidding, it's Appendix D of one of the supplements) explaining how LLMs will ackshually turn out to be exactly like those old bots so nobody needs to do any of that pesky updating of beliefs. To put it bluntly, I do not find this sufficient.

Mind you, there's still significant risk that AI is enough of a shock to civilization/the economy that everything breaks down and the future is horrible. But if you're planning your life around the omnicidal-FOOMing-bot scenario, I really think you're trusting the wrong thought leaders. There's a very large chance you'll still be around in a decade; don't do anything stupid assuming this won't be the case.

Yeah, except in the (IMO rather unlikely) chance that treason charges manage to stick, he basically has to retire to sleep on a big pile of money with many beautiful women. Sucks to be him! (And in the event of a divorce he'll only have half of infinity dollars instead of infinity dollars.)