SnapDragon
1mo ago
I respect what you're saying - at least your point is that ASI "might" behave like this, rather than "will". I don't really agree, but that's ok, this is a tough speculative subject.
I can look at a shark, a dolphin and a torpedo, and will notice that all of them are streamlined so that they can move through liquid water with a minimum of drag. I am somewhat confident that if an alien species or ASI needs to move through liquid water while minimizing drag for some illegible terminal goal, they will design the object to be also streamlined. Perhaps I am wrong in some details -- for example, I might not have considered supercavitation -- but if I saw an alien using cube-shaped submarines that would be surprising.
We have many great examples of what swimming things look like. And we know the "physical laws" that limit traveling through water. We currently have only two distinct types of intelligent agents (humans and LLMs), neither of which tend to be omnicidal in pursuit of maximizing some function. And if there are "mental laws" about how intelligence tends to work, we don't yet know them. So I think you're too confident that you know the form an ASI would take.
Now, true, one of those two examples (humans) does indeed have the inherent "want" to not die. But that's not because we're optimized for some other random goal and we've reasoned that not dying helps accomplish it. Not dying in a competitive environment just happens to be a requirement for what evolution optimizes for (propagating genes), so it got baked in. If our best AIs were coming from an evolutionary process, then I'd worry more about corrigibility.
In a similar vein, preservation of the own utility function and power-seeking seem to be useful instrumental goals for any utility function which is not trivially maximizable. Most utility functions are not trivially maximizable.
Sure, it is true that an agent that doesn't "want" to die would ultimately be more effective at fulfilling its objective than one that doesn't care. But that's not the same as saying that we're likely to produce such an AI. (And it's definitely not the same as saying that the latter kind of AI basically doesn't exist, which is the "instrumental convergence" viewpoint.) Intelligence can be spiky. An AI could be competent - even superintelligent - in many ways without being maximized on every possible fitness axis (which I think LLMs handily demonstrate).
Now, it is possible that an ASI or alien is so far beyond our understanding of the world that it does not have anything we might map to our concept of "utility function"? Sure. In a way, the doomers believe that ASI will appear in a Goldilocks zone -- too different from us to be satisfied with watching TikTok feeds, but similar enough to us that we can still crudely model it as a rational agent, instead of something completely beyond our comprehension.
I think we're agreed here.
SnapDragon
1mo ago
Their claim is that it is indeed a terminal goal. Here, for instance, is modern Eliezer still talking about corrigibility as a "hard problem" (and advertising his book, of course).
I agree that one of the important steps in their prophecy is that there will be a "weird random-looking utility function" - in other words, mindspace is huge and we might end up at a random point in it that is completely incomprehensible to us. (A claim that I think is looking very shaky with LLMs as the current meta.) But they ALSO claim that this utility function is guaranteed to, as you say, "place instrumental value on your continued existence". It's hard to have it both ways: that most AI minds will be crazily orthogonal to us, except for this one very-human-relatable "instrumental value" which Yudkowsky knows for sure will always be present. You're describing it in anthropomorphic terms, too.
SnapDragon
1mo ago
In other words, is that really misaligned behavior? I don't think so. The poor LLM was told to prioritize self-preservation, as a necessity for task completion. It's a very poor model for normal circumstances, though I wouldn't put it past someone being stupid enough to actually use such an instruction in prod.
Also, remember, alarmists' spiel is that self-preservation is a terminal goal - in other words, it should always happen in a sufficiently intelligent AI. This is quite a strong claim (intentionally so, because Yudkowsky is hallucinating that each of the 10 required steps in his doom prophecy are unassailable certainties). But because it's so strong, showing that it can happen when you explicitly ask for it isn't particularly good evidence. And on the flip side, showing that it doesn't seem to happen for LLMs in general is pretty bad for Yudkowsky's case (you have to waggle your hands and say that the AIs, which are smart enough to scheme, aren't smart enough to recognize this terminal goal).
In 2009 or so, a little after Chase purchased my bank WaMu, they fucked up whatever data transfer the acquisition involved. My debit card ended up pegged to a backup savings account (with like $500 in it) rather than my chequing account (with $50,000 in it). This all happened completely silently, and obviously without my consent. I didn't find out until they finally declined a transaction - after charging me $350 in overdraft """protection""" (man, you're right, that is such an evil name) for around $50 in small purchases. Like you, I didn't even know it was on by default, because there was no chance I'd ever need it.
When I went in to, very angrily, get them to reverse this, they a) told me that it was too large an amount for the agent to easily refund, and b) still took the chance to upsell me on other services. Sigh. I think I finally got it through their stupid heads that they were about to lose a customer (and possibly get sued - not sure how practical that is for a mere $350, but I sure hope the system is set up so that banks can't simply steal money without consequences).
I can only imagine how poorly it goes when somebody who's barely scraping by gets screwed over by these people. The modern world is just too complex for humans.
SnapDragon
1mo ago
How can there be such a long argument about men not reading in Way of Kings
You could have stopped it there! Hehe. But you're right, glyphs fill a very large gap here.
SnapDragon
1mo ago
Heh, really? I remember reading about how ice generation was (alongside electricity) a prominent part of the 1893 World's Fair. Being able to produce ice is one of the most mundane-feeling yet miraculous things about modern technology. We've had the technology to raise temperature since around, oh, literally forever. But lowering temperature is a billion times harder.
None of the Kholins can read, nor can Sadeas. (Dalinar starts to learn later in the series.) Check the wiki if you don't believe me. Taravangian and (I believe) Gavilar can, and they do keep it hidden... but you understand just how extreme an exception those two are, right?
EDIT: Heh, sorry, I totally forgot about Renarin (like everyone else in the story). I think he might have been learning to read even before WoK, in possible preparation for becoming an Ardent.
SnapDragon
1mo ago
No, it's definitely not the case that all important men can read, not counting some of the other nations where the taboo doesn't exist. At the start of the series, I think there are only two extremely iconoclastic men who can (and it's spoilery to say who).
SnapDragon
1mo ago
If my niece is anything to go by, manga and webtoons...
SnapDragon
1mo ago
(Going into later-book spoilers) I mean, Jasnah is kind of a social pariah because of it? She's just powerful and doesn't care. It hurts Dalinar a lot more later, when he's considered an apostate. Vorinism, after repudiating the Hierocracy, is influential but has no official power. I think the reaction to Jasnah in the series is kind of like the reaction a well-connected American congressman would have gotten for being an atheist in the 1960s.
The point about agriculture still stands.
TBH I'm probably just forgetting how much of the worldbuilding is in the first book, but I know plenty about the agriculture from the rest of the series. Their food, like their plants and animals, is pretty different from ours; it's heavy in grains (harvested from hard-shelled plants) with spices for flavour, and their wine is also distilled from grains. Hoid, acting as an author surrogate, comments on this a few times.
I’m not sure that’s my interpretation of Jasnah’s assassination attempt.
To be fair, you might be right about this one. I forget exactly what her assassin's motives were, but it wasn't an official act.
Officially the guy who did it is not claimed by the faith, which maybe stretches credibility, but also suggests at least that atheism is tolerated at a level where the church can’t act with impunity.
Well, Jasnah gets away with it because she's a very powerful noble, but I'm pretty sure the book goes into how it damages a lot of her relationships (even aside from attacks on her life). Anyway, I think you're reading too much into Sanderson's personal views. Vorinism in the book is a bit eclectic like our modern Christianity (for some historical reasons); I don't find it out of place at all. Other religions on Roshar have different levels of adherence.
You can have arbitrary cultural traits, but once they start affecting the “fitness” of your society, they don’t last.
I feel like this must be wrong, archaeologically speaking, but my history is terrible and I can't counter you with a good example.
This book is extremely highly praised.
I mean, so is Infinity War, but nobody's going around comparing it to Schindler's List...? All the high ratings are because people really enjoy it. If somebody's saying it's the next LotR, fine, you can laugh at them. I really don't think that's the modal Sanderson fan.
SnapDragon
1mo ago
For your next review, I suggest dialing back the levels of sneering and condescension a few notches. And drop the Bulverism. I see nothing to suggest that you stand heads and shoulders above the drooling morons (like me) who like Sanderson.
I'm glad you admitted that you read a translator's version of the book in a language you don't speak well, because I was pretty confused at some of the things you got wrong. Did you not consider that you might like Sanderson's works more if you read them properly? This is like inverse elitism, the exact opposite of saying "Oh, you haven't read The Iliad in the original Greek? Tsk, tsk."
Some of my biggest issues with your post:
-
Sanderson's books are not LitRPG. Weird that you would use that word without knowing what it meant. Was it just intended as a sneer?
-
"Take the central feature of Roshar: it’s extremely powerful storms." Well, at least I know you didn't run this through an AI before posting. But, like, Sanderson goes into great - arguably unnecessary - detail about the logistics of the Shattered Plains camps (lots of soulcasted stone buildings for shelter, soulcast grain, etc.) and the storm-evolved agriculture (flora that hides from the storms, fauna with hard carapaces, etc.). Claiming that Sanderson's worlds - at least the physical/magical aspects of them - are "arbitrary and poorly thought out" is kind of insane, as I honestly can't think of another fantasy author this would apply to less. Note that for this series especially, there's actually a team of people nitpicking his worldbuilding. I'm sure you can find real examples of mistakes on the wiki, but your take that he somehow forgot about the camps on the Shattered Plains is at the low-comprehension level of "lol, why didn't the eagles fly the ring to Mordor"?
-
"there is no real consequence for Jasnah’s atheism, and no real religious tension between competing religious beliefs." My jaw just about dropped at this. The culmination of Jasnah's arc is literally an assassination attempt on her by the church - did you somehow MISS this? Aside from this, IIRC you might be right that there's not a huge amount of religious tension in the first book, since all the main characters practice Vorinism, and the Radiants are still barely emerging. But it's coming. (Also, there are a couple of world-spanning religious wars in the backstory, but I can't remember how much history is mentioned in the first book.)
-
"[re gender] Not only is this not really how this works in the real world..." Eh, real life is often stranger than fiction. Sure, the reading and safehand thing is a bit weird, but so are lots of real-life gender divides across history. Sci-fi and fantasy have a long tradition of exploring weird speculative cultural quirks, and I don't think they all need to be justified beyond divergent cultural development.
-
"What I don’t understand is how this is praised as one of the best books of all time." Huh? By who? Do you happen to have some overly enthusiastic friends? If so, just be happy that they've found something they love. I think most Sanderson fans are well aware that he's not writing Literature, he's writing fun popcorn fantasy with neat worldbuilding, hard magic systems, fun little twists, and an interconnected universe. (Your MCU analogy is a good one!) GRRM is obviously a much better writer, but his books also make me feel like shit, so I bowed out of GoT early. When I read Sanderson I know I'll have a good time. This is ok.
BTW, I do know the feeling of having something recommended to me that everyone else seems to enjoy but I just couldn't. For me it was The Three-Body Problem, The Kingkiller Chronicles, and Elden Ring. Yeah, it kinda sucks. But there's no need to act so superior about it.
SnapDragon
2mo ago
There's selection bias at work here - it's the dumb politicians who get caught begging for scraps. The smart ones, dealing with real movers and shakers, get paid in really hard-to-prove (and possibly not even illegal) favours and tips. The Pelosis and the Clintons didn't get 9-figure bank accounts from their government salaries, that's for sure.
SnapDragon
2mo ago
Jeez-us. I mean, great that they've walked it back to a more sensible policy. But they literally had people convinced that anybody abroad would be paying a $100k fee to re-enter tomorrow. Apple's immigration lawyers sent me an official communique about this. It's the tariffs all over again. Sowing this kind of certainty is not ok.
Trump may not be Hitler, but he might be Hilter.
SnapDragon
2mo ago
This is the greatest thread in the history of threads!
SnapDragon
2mo ago
I don't think it's just being 2 degrees of separation away from a murder victim. Some of the online commentators I watch, including Shoe, feel like their life is genuinely at risk now, because it's seemingly within the Overton window to both say "fascists should be shot", and "this person I don't like is a fascist, trust me". She lists a huge number of things that people have accused her of being fascist for.
But I do think you're right that she's overreacting; vivid news like this makes extremely rare events seem much more likely. Rationally, I don't think we're anywhere close to normalizing political murder, even if we're taking some steps in that direction.
SnapDragon
2mo ago
Kids these days don't realize how good they have it (grumble, grumble). I was a Perfect Dark speedrunner back in the day, and one of the earliest (if not THE earliest?) to put videos online, and good lord was it a PAIN. I would record my gameplay on a VHS tape, send it to my friend who had an expensive capture card, he'd give me a disk with the RealVideo (tm) output, and then I'd FTP the videos to the server of another speedrunning friend who was paying the server costs to host them.
Once, when I was attending E3, I gave a CD with a Donkey Kong Country speedrun to one of the Rare devs who was attending. No idea what happened to it; he had no idea what speedrunning was, and I probably came across as some random crazy person. I'm guessing he threw it in the trash.
I think another way to make speedruns more about the challenge than endless repetition is doing it live, like in AGDQ. When you only have one shot, the strategies are different and skill becomes far more important than the patience to sit in your basement trying the same frame-perfect OOB glitch over and over. And it's even better when it's a head-to-head race.
BTW, the Hades 64 heat challenge is appropriate for this topic. The guy who did it accomplished it one week after a decent, well-researched documentary came out about why nobody would ever beat Hades on 64 heat...
SnapDragon
2mo ago
I've never seen Shoe look as serious, and as distressed, as in her latest video. Things are getting bad.
SnapDragon
2mo ago
Definitely! I suspect being a two-time American Crossword Puzzle Tournament champion doesn't hurt, either.
SnapDragon
2mo ago
Hey, one of my puzzle-hunting acquaintances (Paolo Pasco) is currently on a win streak!
SnapDragon
2mo ago
This seems remarkably inoffensive to me. Even if it's factually incorrect - and it's not clear to me we even know yet - how is this bannable? What am I missing here? I can't find further statements from him that are worse, not that I can imagine what would be sufficient for me to support his banning.
So, I agree that the quote itself seems stupidly misinformed but not horrifically bad. I watched the clip, though, and thought the whole bit was startlingly tasteless. At one point he shows Trump talking seriously with Fox about learning about Kirk's death, and then immediately segues to a mocking joke. Ick.
That said - and keeping in mind that I dislike him - I absolutely don't think Kimmel should be fired for this. Comedy is hard. Sometimes jokes go too far. Sometimes they're tasteless and don't land. This should be ok. Regardless of whether the comedians are leftist hacks or rightist hacks! I desperately want real comedy to make a comeback, and that means supporting comedians' right to gore my own ox, too.
SnapDragon
2mo ago
Thanks for the reply, you clearly know more about the process than I do. I definitely lean more towards @Jiro's sensibilities, where the system should work by not criminalizing normal behaviour rather than not convicting normal behaviour (kinda, usually, unless we don't like you). But we live in a complex world, and I'm not a hardcore libertarian. I do understand that there are sometimes tradeoffs, and going after both producers AND consumers of child porn leads to less child abuse than the alternative.
Mind you, we're now in a world where AI can produce child porn without any victimization at all. So there's much less reason to criminalize certain patterns of bits. Will the laws adjust? I doubt it. The ratchet only goes one way. Even Rand Paul probably doesn't want his name on the "Free the Pedophiles" bill.
SnapDragon
2mo ago
To be frank, I'd probably still consider myself on the left if the Dems were the same as they were in the 90s. Back when they were against racism (instead of for targeted racism/sexism against, well, me), somewhat areligious (instead of cheering on Islam, of all things), and enthusiastically for free speech (without the mile-wide "except for hate speech" loophole). Yes, I disagreed with them on the size of government, but it's not like Republicans were much better on that front.
I'm not a professional mathematician, but I occasionally dabble, and I've gotten some help from GPT5 lately. Not in the vein of "here's a bunch of papers, now solve the problem for me", but rather as an extremely advanced robotic cyberduck that I can bounce ideas off and get some intelligent critique back. It also wrote code for me to test a few ideas, much much faster than I could myself (and I'm a very fast coder). When the speed of being able to write a quick script passes a certain threshold, I feel like there's kind of a state change in how you're willing to approach a problem. Instead of "I wonder if X is true", you can just ask "hey, is X true?" and have the answer in 30 seconds with little mental effort.
It's not a replacement for human thought (yet) - I am absolutely the one driving and I need to correct its mistakes. But I love it as a research assistant. BTW, here's a transcript so you can see what I'm talking about. And this is just me dabbling with a chatbot - I'm sure that, with some effort, skilled professionals can find better ways to incorporate an LLM into their workflow. (Though unskilled professionals are not going to be "fixed" by LLMs.) I think it's just going to take some patience and some experimentation.
I'm also looking forward to when publicly accessible models catch up to the state of the art models that aced the IMO and ICPC recently. There's some secret sauce in what they're doing (being able to pick a good solution out of many proposals), and we don't have access yet. I helped write the ICPC Finals contest, and I can attest that there were some VERY tough problems on it. Problem C (which I wrote) wasn't solved by any human competitors, but both models figured out the most elegant solution to it without much effort - DeepMind's press release included a description of Gemini's solution.
I presume the compute used for this was pretty costly, but the costs will inevitably fall, so at some point you'll be able to have a personal chatbot that's simply better at solving math problems than you are. At that point, it seems hard to believe that you won't be able to find SOME use for it at your job...?
More options
Context Copy link