roystgnr
23d ago
I just want to make sure it has qualia and isn't a Chinese room.
"In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children." - Nick Bostrom
I'd also add some preferences regarding population and personality and such, but "do our successors have any intrinsic value or not" does seem to be the first and most important criterion to have!
However, I'm confused by the use of the phrase "make sure" here. Unless you're expecting to be uploaded, and you're confident that the idea of a "p-zombie" is incoherent (which I'm guessing you aren't, given the Chinese room reference), what observations could give you any sense of surety here? Today's LLMs can pass Turing tests, which used to be our "fine, they're sentient now" criterion, but their lack of "medium-term" memory and they fact that they still can "slip" in ways that make them seem non-sentient makes us think in hindsight that our criterion was just inadequate, and yet we haven't really found anything to replace it. If tomorrow's LLMs never slip, does that mean they've become sentient, or does that just mean they've become better at faking it?
roystgnr
23d ago
Even "overestimated" probably overstates things, in that it suggests that he got the magnitude wrong but still got the direction of the effect correct. I suspect it's more likely that stop-AI bombings will have roughly the same effect on AI risk that anarchist groups' bombings and murders a century ago had on government overreach.
roystgnr
23d ago
Taking inflation into account, that's only a little more than I was getting paid by my university just to work on my PhD. From an independent company who'd have you working on problems with more immediate benefit to them than "it makes the University stats look better" and/or "we could conscript him to teach if a prof gets sick or quits suddenly", it doesn't sound competitive at all until you consider the equity ... and equity in a startup is like a lottery ticket: even if the game isn't crooked, your ticket might make you fabulously wealthy but it's more likely to be worthless.
On the other hand, the job market does seem to be kind of awful right now. It might not be crazy to take something here to avoid resume gaps and build more experience while job hunting elsewhere, and if you're finishing up your PhD at the same time then maybe that's enough to prevent the typical "what was your last salary" question from making subsequent employers lowball you too.
On a side note that probably belongs in Culture War - the Baltimore homicide rate is now the lowest its been in nearly 50 years, after dropping more than 60% in 3 years! Wow! That still leaves a crazy high rate (my advice after my daughter's Johns Hopkins application and some crime map study: "you're not likely to get shot unless you go a mile south, or east, or southwest ... north looks nice ... I can see why they're medical specialists ...") but there's now like 500 fewer dead people than the 2015-2022 rate would have predicted, and that's pretty great.
roystgnr
23d ago
there has basically never been a mathematician producing valuable maths older than that.
Yitang Zhang didn't prove that the lim inf of the prime gap was 2 (which would have verified a 150 year old conjecture), but he was the first to prove it was finite, at age 58. Listing and Moëbius were in their 50s when they formalized the idea of non-orientable surfaces (which I would consider to be literally "producing" math rather than just "solving" it). The first version of the Weierstrass Approximation Theorem (which led to whole fields of the most economically valuable results in mathematics, in my biased applied-math+engineering opinion) was proven when Weierstrass was 70.
But, damn, are they the exceptions who prove the rule, so long as we leave that "basically" qualifier intact? Kolmogorov was in his 50s when he solved Hilbert's thirteenth problem, but that was in joint work with a 19 year old student. Euler, Gauss, and Cauchy were doing great work in their old age, but arguably only after doing greater work as younger men. Searching for mathematical discoveries by importance and then looking up age (rather than searching specifically for discoveries made at older ages), the bulk do seem to be between 25 and 45.
I wonder if the trend is moving older (because things that were groundbreaking discoveries 300 years ago are basic undergrad background today and you need to learn much more to get to the cutting edge) or younger (because subsidized institutional math research gets more output but at the cost of making older mathematicians spend all their time teaching and mentoring and writing proposals and hiring and so on, while their grad students and postdocs are the ones who can actually focus on the work).
roystgnr
25d ago
Well I never said that
That was supposed to be the generic you in context; sorry for the misleading ambiguity.
roystgnr
26d ago
He does want what is best for Ankh-Morpork, but what is best for Ankh-Morpork is Him. Common failure state of Tyrants in general.
Yeah, but the trouble is that the other failure state is "when the tyrant is gone, people fighting over the power vacuum tear the place apart". Monarchy (with one well-defined rule of succession or another) was actually a valuable social technology at one point, and it's one that Ankh-Morpork doesn't currently have access to!
It's entirely possible that, even if Vetinari decides that Carrot (or someone else, considering Carrot's objections) would be a better leader, he still doesn't see any way he can name a successor and retire or die without that successor being under more threat than he was and the city needing a successor to the successor in short order.
It's a shame we never got to see how Pratchett would have had things all turn out. I like the "Vetinari tries to leave von Lipwig in charge, and Moist weasels out of it by hastily inventing democracy" fan theory, personally. That's not a story, though; you could describe most of the Discworld plots that plainly and unimpressively. The stories didn't get great until you got down into the details.
roystgnr
26d ago
Vetinari is possibly good in a utilitarian sense, just not in most virtue or deontological senses. The vibe I get is a cynical "it's okay for me to be bad enough to prevent my being replaced with something even worse". He does seem to be grooming people like Vimes and von Lipwig to actually make things better, but even his attempts to delegate strike me as a mix of pessimism ranging from "I could retire before I'm killed if things go well enough" to "I could need scapegoats if things go poorly enough"
roystgnr
26d ago
I don't think it's sexist or racist to believe that men are more likely to be violent
Replace "violent" with "at the highest end of STEM aptitude" and you just got fired from the presidency of Harvard. These are somewhat overloaded words but the revealed descriptivist definition seems to include any belief in inherent differences that have clear moral valence, at least if the direction of some of those perceived differences can be interpreted as "punching down" the progressive stack.
or that Muslims are more likely to be violent.
Well, this one's straightforwardly not racism because "Muslim" isn't a race, but it is classic "Islamophobia".
It's kind of weird that we don't have a general "-ism" word to refer to religious prejudice or religious intolerance, but I guess religion is the point at which the "everybody's the same regardless" theory breaks down so badly that nobody wants to say religious prejudice is always unwarranted? You may believe Islam in particular isn't inherently more violent (it tried to be more resistant to the "evolution" that @Rosencrantz2 points out is common to religions, yet in practice Muslims' attitudes toward violence do vary greatly from century to century and place to place) But, are you going to extend that charity to every religion ever? That's a good way to find yourself visiting the People's Temple for a free glass of Flavor Aid.
roystgnr
26d ago
That all makes sense, thank you.
I still feel like the list of sanctions on Pakistan is damning the idea with faint praise, though. The US cut off military and non-humanitarian foreign aid (not even a trade embargo!) once Bush I could no longer swear that Pakistan didn't have nukes, then 8 years later they definitely did have nukes and we also cut off financial lending/underwriting, then a few years after that it was inconvenient for the one country who was actually pushing, and now we're all fine? Becoming a nuclear state isn't completely costless, but Pakistan still doesn't seem to have been punished as much for nukes as Afghanistan was for box cutters.
roystgnr
26d ago
next-token prediction really isn't the kind of intelligence that we wanted to develop, but it's what we discovered first.
(Cries in Yudkowsky)
perhaps at some point we'll discover a different framework for AI that better matches our own sapience at lower cost.
I think this is what makes "FOOM" still something of a risk. What are the odds that we really discovered the most computationally efficient implementation of intelligence on the very first try and step one really was "just download the internet and try to compress it"? When we solved problems like magnetohydrodynamics simulation, we had some much more clever initial ideas, yet we still managed to improve them another order of magnitude (just software; another OOM in hardware) in each of the next few decades. There's still a fundamental limit to how efficient any particular algorithm can be, but it's not out of the question that, once we have a ton of artificial researchers that don't need to be handheld on every short task, we'll get a similar 1000x sort of speedup, much sooner.
better matches our own sapience
If we just match our own sapience, then the hallucination problem really can't be solved, rather than just mitigated. Humans still do that shit all the time. I once missed a problem on a Differential Equations quiz because I evaluated "1 × 2" as "3" in an intermediate step.
roystgnr
27d ago
Right now I think coding models are at their most powerful when being used as a force multiplier for human experts.
This matches my experience. Today Codex generated a PR for us that fixed a bug, but it missed three more instances of the same bug, broke some functionality with the fix, and didn't have a hint of the work we'd need to do to reenable that functionality for existing users while migrating them to a newer implementation with wider compatibility.
BUT: all that extra work won't take nearly as long as it would have taken the human Codex user to find the initial bug. Still a big win.
At this rate, though, how long will it be until we don't need the human user? Centaur chess lasted maybe a decade, being generous, but at this point last year AI had only basic "much better search engine" utility for me, and at this point two years ago it was downright counterproductive to try to sort out real answers from hallucinations. Where will we be in another five years?
You don't complete the sentence "The answer is" with "oh wait never mind I don't know".
No, but somehow these days they're tuning their final models to get to "I don't know" anyways. Maybe they're not just glorified autocomplete? 10 months ago was the first time I got an LLM to admit it couldn't answer a question of mine (although it did still make helpful suggestions); not only did the other models back then give me wrong answers, IIRC one of them went on to gaslight me about it rather than admit the mistake. (two years ago this gaslighting would have been the rule rather than the exception) IMHO that "I don't know" was the exact point at which AI started to have positive utility for me. Sometimes an AI still isn't helpful, but it's at least often worth throwing a problem at now, not a waste of time.
roystgnr
27d ago
a bunch of informal pressure to ensure this doesn't happen to the extent possible
This is a good point. The trouble is that world leaders act like they're big Causal Decision Theory fans, and once a state has nukes it's hard to go back in time to make that not have happened, so whatcha gonna do? We try to keep ICBM tech from leaking to Pakistan, but we hardly turned them or India into pariah states for having the warheads. Maybe Iran would get worse treatment because they signed on to the Non-Proliferation Treaty and would be violating it whereas non-signatories weren't?
Enough intel is public that we know Iran had a bunch of nearly bomb grade enriched uranium, but that they just stopped at that point and made no further effort to weaponize.
Ignorant question: how confident are we of that? It looks like Iran fired two missiles at Diego Garcia, at more than double the range of anything we publicly knew they had (and if we knew privately that they were violating ICBM restrictions, that would have been a great cassus belli to bring up to Europeans uninterested in joining this war, so I'm betting we didn't), in which case they've at least been managing to keep some aspects of their weapons development programs secret.
roystgnr
27d ago
FWIW, these days you can usually ask about the seemingly-retarded choices and get either an explanation or a correction-plus-apology for each. I use AI to self-study math, and I still catch it making errors these days, but no more often than it catches my errors, and pinning down which case is which usually only takes a little extra back-and-forth. The reasoning models these days are much better than when some of them would just double down and try to gaslight you about their errors ... Whoa, according to my logs that was only a year ago. Hell of a roller coaster we're on...
roystgnr
27d ago
Why were those guys even carrying around an extra kidney they didn't need, anyway?
roystgnr
27d ago
"Density" definitely becomes a quality to look for in a game rather than one to avoid. I remember reviews criticizing some RPG for only having 20 hours of content after 30 became the de facto standard, and I agreed, but these days? You tell me I'm not going to get closure for 20 or 30 hours? If it's not one of the best games of all time, count me out; if it is, I swear I'll try to get around to it in 5 or 10 years. I'd love a list of modern games that instead follow in the footsteps of Portal 1 (3-4 hours) or at worst Portal 2 (8-9?) and likewise manage to pack a full satisfying game into that length of time.
roystgnr
27d ago
edifying in the same way a good nonfiction book is.
Well, the good nonfiction book is typically denser.
I've been a fan of space flight my whole life, nearly studied ASE before starting off as a MechE instead, but I never understood why "porkchop plots" had their weird double-lobed shape until the first time I found myself having to cope with interplanetary plane change maneuvers in Kerbal Space Program. So, yeah, on the one hand, actual advanced practical (well, to JPL engineers anyway) knowledge ... but on the other hand, Steam tells me I've put 320 hours into KSP over the last 10 or 15 years, and I'll bet I could have gotten more knowledge out of a good orbital mechanics textbook in a quarter of the time.
roystgnr
27d ago
The only cost to an autocracy getting nuclear warheads is that, if you don't stay personally in charge of them, your successors can be as tyrannical as they want and nobody will come save you from them. This is more than counterbalanced by the benefit that, if you do stay in charge of them, nobody will come try to "save" anyone from you. North Korea won't be getting the Venezuela or Iran treatments any time soon.
Getting highly-enriched uranium without continuing on to turn them into warheads, on the other hand, just pisses everybody off without giving you any leverage, and the next thing you know your successors are in charge anyway. Even if you have a weaker bomb program and give it up before the airstrikes escalate, moving far enough in that direction may already have crossed the "sodomized to death by a bayonet eight years later while the world chuckles" point of no return. This is just not a place where you stop your nuke program because your political calculations are going well; it's a place where you stop because your engineering calculations aren't going well enough. A successful test explosion is a "Get Out Of Jail Free" card; a test fizzle is a "Kill Me Now Before It's Too Late" request.
roystgnr
28d ago
in a replacing doctors scenario you'd need to be getting it right 100% of the time with no second check
Which doctor manages that?
Yeah, I know, that sounds like an insult/joke, but estimates of iatrogenic death rates in US hospitals are at minimum 20k/year out of 700k, which means that in even in high-stakes scenarios doctors and nurses are only at 97% for "getting it right enough not to kill someone"; fully "getting it right" would be a much higher bar and lower success rate. All the AI has to do to replace skilled workers is get more reliable than they are.
You're surely right that AI in medicine isn't as good a replacement for human workers as AI in fields where checking results is easier than producing them ... but it's perhaps similar to AI for self-driving cars: stringent requirements and potentially-lethal consequences, but the AI still doesn't have to be perfect to be an improvement, it just has to be better than the typical human competition.
roystgnr
28d ago
I think the median 0day the NSA exploits is one they found or bought and not one which they made some US company insert on purpose.
You're probably correct, but don't forget about the (probably small, but not null) class of exploits that they simply trick US companies into inserting. The NSA has a wide range of strategies. They paid RSA to use their exploitable Dual_EC_DRBG, for instance, but apparently that was mostly to buy enough credibility to get it called "the standard" and adopted freely by other crypto companies too.
Even their work with DES was a mix of white-hat (they knew about a vulnerability and pushed for changes that they secretly knew would eliminate it) and black-hat (they pushed to drop the standard key size from 64 to 48 bit, then settled for 56, because they knew they had the compute to brute-force those) security, and the only "made some US company insert on purpose" there was legislative, for a brief period in the late 90s when companies were only allowed to export encryption software with 56-bit or shorter keys.
roystgnr
30d ago
Computerized car engines give you more efficiency, more power, less pollution, and more reliability than mechanical ignition and injection systems did.
Computerized driver controls I'm much less sanguine about. Nothing I might want to fiddle with while watching the road should be controlled by a modal UI, much less by a touch screen, rather than by a knob or button whose function is determined by a shape and position I can actually discern by touch.
get data from their cars and change the cars' behavior without having to modify hardware?
Well, this is perhaps the source of the design problems, not just the design decision, isn't it? Some of the common examples of risk compensation are claims that car drivers take more risks when they know they have anti-lock brakes or seat belts partially protecting them from the consequences, but software producers, including car software producers, also have incentive to take more risks when they know they have automatic patch application partially protecting them from the consequences. Do you really need to fix all the bugs before launch now, or can you just fix the worst of them and then try to get to more of the rest before buyers get too pissed?
If my theory here is right, then computerized car engines could actually get worse as cabin computer connectivity gets more popular. If your buyer can't do an ECU firmware update without going into the shop, they're going to be pissed if they ever need an update, and you'd better make sure that engine computer is solid from day one, whether or not there's a bug in the radio UI. But if you have an internet connection that lets you slip an ECU firmware update in without the buyer even noticing? Getting software solid is expensive, and you could probably save a lot of time and money by just getting it mostly solid and then waiting for the diagnostic data and the bug reports...
roystgnr
30d ago
I'm typing this from inside my solar-panel-bedecked house in Texas, so don't take it too critically, but Texas is among the best spots for solar, not the typical spots. I get 300 days a year of good sunlight, and most of my residential energy demand is coming from A/C requirements that spike only a few hours after the solar supply does, so people here can get away with few batteries ... or if you rely on the grid for evening power like I do, no on-site batteries.
If your city doesn't get as much insolation (most of China, for some reason?) the economics look a little different. If much of your home energy demand is coming from a heat pump that needs to be pumping in the opposite direction (what, half the US?), the economics look a lot different. If both are true (half of Europe?) then it's long past time (except for you, France, we're cool) to just go nuclear.
roystgnr
30d ago
That average temperatures across the globe taken over the course of year X, are higher than x-1, and x+1 will be higher than x.
Well, this one's just not true; e.g. 2021 was cooler than 2020. Even x to x+10 failed for 1998 vs 2008. I'm not sure if we'll ever see a decadal drop again without some major change, but the trend is still only something like 0.025C/year, and year-to-year variation of ~0.1C still regularly swamps that.
Less snarkily, that the models popularised over the last 50 years or so have been mostly correct, and all show a warming trend.
Remove "popularised" and this one's true. But the way we (well, the news media and the most salacious academics) popularize a model is to take whatever the most extreme possibility is at one p<0.01 end of the predictions, turn that into a "it will never snow in Britain again!!" headline, and so convert tomorrow's credibility into today's paycheck. Current measured warming since 1990 is pretty much right on the "Business-as-Usual"+"Best Estimate" line from the 1990 IPCC report, which puts it way above the "weather just changes, it'll probably revert to the mean again" null hypothesis, but "we're up nearly 1C and it's still accelerating" feels anticlimactic to people who vaguely think they remember that coastal cities were supposed to be flooding or something by now.
roystgnr
1mo ago
It's not that her decision is more dumb or more awful, or that she's expected to have more agency, it's that it's dumb and awful in a more self-destructive way, so we'd hope to be able to skip the step where we figure out what incentive is supposed to prompt the use of that agency. The man here is obviously in vastly greater need of improvement, but "Wouldn't you like to not get to abuse anybody, not get to control anybody, and not get to be a drunk?" seems like a harder sell, at least for the type of person for whom it's applicable, than "Wouldn't you like to not get abused, not get controlled, and not get stuck with a drunk?"
roystgnr
1mo ago
Damn it, I'm one Prime Minister off. I meant Boris Johnson.
Shwarzenegger's affair partner was 6 years younger than his wife, though I wouldn't say hotter. After Gavin Newsom's divorce he dated a woman literally half his age, but the woman he married was only 7 years younger. I'm afraid I also misremembered a bit about Newsom: the "affair" he was part of might not have started until after he was divorced, in which case the only infidelity was that the woman was married to ... well, Wiki calls Alex Tourk "Newsom's close friend", but I feel like the rest of the sentence disproves that description.
So ... which is it? Populist demands are easily converted (by both sides of the aisle!) into protectionist policies that set up parts of the economy for rent extraction. "You can only build more housing here if it's economically 'inclusionary' enough" gets predictably turned by reality into "you can't build more housing here" and drives up the price of the grandfathered (often literally!) housing stock. People want to "drive housing prices up for people who own their homes" while also making housing prices affordable for people who don't, but that just doesn't compute.
are another example. The ACA caps insurance company profit as a percent of premiums, a policy at least populist enough for Obama to brag about it ... and a policy that inadvertently sets up a huge conflict of interest when insurers are trying to figure out what they should pay out.
Ironically, this sort of "cost-plus contract" malincentive was also fixed in part by Obama, in the context of NASA procurement, when he supported and extended the Commercial Resupply Services contracts and then went beyond them with the Commercial Crew program, in both cases paying for purchases where the seller actually could make more profit by producing results more cheaply. For now we still have to burn $4B a pop for SLS when we want to send humans outside Low Earth Orbit, but we can replace it with a $180M Falcon Heavy launch for things like Europa Clipper.
More options
Context Copy link