SnapDragon
12d ago
There's also the concern of what kind of suffering a post-singularity society can theoretically enable; it might go far, far beyond what anyone on Earth has experienced so far (in the same way that a rocket flying to the moon goes farther than a human jumping). Is a Universe where 99.999% of beings live sublime experiences but the other 0.001% end up in Ultrahell one that morally should exist?
SnapDragon
12d ago
I'm inclined towards your skeptical take - I think we as humans always fantasize that there are powerful people/beings out there who want to spend resources hurting us, when the real truth is that they simply don't care about you. Sure, the denizens of the future with access to your brainscan could simulate your mind for a billion subjective years without your consent. But why would they?
The problem is that there's always a risk that you're wrong, that there is some reason or motive in post-singularity society for people to irreversibly propagate your brainscan without your consent. And then you're at the mercy of Deep Time - you'd better hope that no beings that ever will exist will enjoy, uh, "playing" with your mind. (From this perspective, you won't even have the benefit of anonymity - as one of the earliest existing minds, it's easy to imagine some beings would find you "interesting".)
Maybe the risk is low, because this is the real world we're dealing with and it's never as good or bad as our imaginations can conjure. But you're talking about taking a (small, you argue) gamble with an almost unlimited downside. Imagine you had a nice comfortable house that just happened to be 100m away from a hellmouth. It's inactive, and there are guard rails, so it's hard to imagine you'd ever fall in. But unlikely things sometimes happen, and if you ever did, you would infinitely regret it forever. I don't think I'd want to live in that house! I'd probably move...
SnapDragon
2mo ago
We shield kids from a lot of complicated real-world things that could affect them. 4-year-olds can have degenerative diseases. Or be sexually abused. Both are much more common than being "intersex" (unless you allow for the much more expansive definitions touted by activists for activist reasons). So I guess schools should have mandatory picture books showing a little kid dying in agony, while their sister gets played with by their uncle, right? So that these kids can be "at peace" with it?
...Of course not. Indoctrination is the only reason people are pushing for teaching kids about intersex medical conditions. Kids inherently know that biological sex is real, and can tell the difference between men and women. Undoing that knowledge requires concerted effort, and the younger you start, the better.
SnapDragon
1mo ago
Let me join the chorus of voices enthusiastically agreeing with you about how jobs are already bullshit. I've never been quite sure whether this maximally cynical view is true, but it sure feels true. One white-collar worker has 10x more power to, well, do stuff than 100 years ago, but somehow we keep finding things for them to do. And so Elon can fire 80% of Twitter staff, and "somehow" Twitter continues to function normally.
With that said, I worry that this is a metastable state. Witness how thoroughly the zeitgeist of work changed after COVID - all of a sudden, in my (bullshit white-collar) industry, it's just the norm to WFH and maybe grudgingly come in to the office for a few hours 3 days a week. Prior to 2020, it was very hard to get a company to agree to let you WFH even one day a week, because they knew you'd probably spend the time much less productively. Again, "somehow" the real work that was out there still seems to get done.
If AI makes it more and more obvious that office work is now mostly just adult daycare, that lowers the transition energy even more. And we might just be waiting for another sudden societal shock to get us over that hump, and transition to a world where 95% of people are unemployed and this is considered normal. We're heading into uncharted waters.
SnapDragon
1mo ago
Please post an example of what you claim is a "routine" failure by a modern model (2.5 Pro, o3, Claude 3.7 Sonnet). This should be easy! I want to understand how you could possibly know how to program and still believe what you're writing (unless you're just a troll, sigh).
SnapDragon
1mo ago
There are plenty of tasks (e.g. speaking multiple languages) where ChatGPT exceeds the top human, too. Given how much cherrypicking the "AI is overhyped" people do, it really seems like we've actually redefined AGI to "can exceed the top human at EVERY task", which is kind of ridiculous. There's a reasonable argument that even lowly ChatGPT 3.0 was our first encounter with "general" AI, after all. You can have "general" intelligence and still, you know, fail at things. See: humans.
SnapDragon
3mo ago
I agree with you, but I'll note that our entire legal system seems to be based on "one weird trick"s, all the way down. That's how they got a felony conviction against Trump for a misdemeanor whose statute of limitations had expired. Unfortunately if the system really wants to get you, they will. I don't know how to fix it, but at the very least let's keep calling it out wherever we see it.
SnapDragon
1mo ago
What the hell? You most definitely did NOT give any evidence then. Nor in our first argument. I'm not asking so I can nitpick. I would genuinely like to see a somewhat-compact example of a modern LLM failing at code in a way that we both, as programmers, can agree "sucks".
SnapDragon
1mo ago
Right, and I asked you for evidence last time too. Is that an unreasonable request? This isn't some ephemeral value judgement we're debating; your factual claims are in direct contradiction to my experience.
SnapDragon
1mo ago
I'm also not allowed to use the best models for my job, so take my advice (and, well, anyone else's) with a grain of salt. Any advice you get might be outdated in 6 months anyway; the field is evolving rapidly.
I think getting AI help with a large code base is still an open problem. Context windows keep growing, but (IMO) the model isn't going to get a deep understanding of a large project just from pasting it into the prompt. Keep to smaller components; give it the relevant source files, and also lots of English context (like the headers/docs you mentioned). You can ask it design questions (like "what data structure should I use here?"), or for code reviews, or have it implement new features. (I'm not sure about large refactors - that seems risky to me, because the model's temperature could make it randomly change code that it shouldn't. Stick to output at a scale that you can personally review.)
The most important thing to remember is that an LLM's superpower is comprehension: describe what you want in the same way you would to a fellow employee, and it will always understand. It's not some weird new IDE with cryptic key commands you have to memorize. It's a tool you can (and should) talk to normally.
SnapDragon
21d ago
SnapDragon
2mo ago
I've had the "modern AI is mind-blowing" argument quite a few times here (I see you participated in this one), and I'm not really in a good state to argue cogently right now. But you did ask nicely, so I'll offer more of my perspective.
LLMs have their problems: You can get them to say stupidly wrong things sometimes. They "hallucinate" (a term I consider inaccurate, but it's stuck). They have no sense of embodied physics. The multimodal ones can't really "see" images the way we do. Mind you, just saying "gotcha" for things we're good at and they're not cuts both ways. I can't multiply 6 digit numbers in my head. Most humans can't even spell "definately" right.
But the one thing that LLMs really excel at? They genuinely comprehend language. To mirror what you said, I "do not understand" how people can have a full conversation with a modern chatbot and still think it's just parroting digested text. (It makes me suspect that many people here, um, don't try things for themselves.) You can't fake comprehension for long; real-world conversations are too rich to shortcut with statistical tricks. If I mention "Freddie Mercury teaching a class of narwhals to sing", it doesn't reply "ERROR. CONCEPT NOT FOUND." Instead there is some pattern in its billion-dimensional space that somehow fuzzily represents and works with that new concept, just like in my brain.
That already strikes me as a rather General form of Intelligence! LLMs are so much more flexible than any kind of AI we've had before. Stockfish is great at Chess. AlphaGo is great at Go. Claude is bad at Pokemon. And yet, the vital difference is that there is some feature in Claude's brain that knows it's playing Pokemon. (Important note: I'm not suggesting Claude is conscious. It almost certainly isn't.) There's work to do to scale that up to economically useful jobs (and beating the Elite Four), but it's mainly "hone this existing tool" work, not "discover a new fundamental kind of intelligence" work.
SnapDragon
4mo ago
It may not be a universally-accepted truth, but it is a scientific truth. We're a sexually dimorphic species. There are plenty of tests which easily tell the two groups apart with 99.99% accuracy, and if you're MtF you'd sure as hell better inform your doctor of that fact rather than acting like you're just a normal woman.
Joe Blow down the street thinks he's Napoleon. So, it's not a "universally-accepted truth" that he's not Napoleon. And maybe he gets violent if you don't affirm his Napoleonness in person, so there are cases where feeding his delusion is the path of least resistance. There's a "fundamental values conflict" there. But it remains an objective truth that he's not Napoleon.
SnapDragon
4mo ago
Yes, the whole theoretical point of academic tests is to be an objective measure of the capacity of students. Because when you go out and get a real job, you have to actually be able to do that job. If these remedial courses aren't necessary for being a psychiatrist, then there should be a path to becoming a practicing psychiatrist that doesn't require them. If they ARE necessary, then lightening the requirements because, gosh, you can't satisfy the requirements but really want to graduate ends up causing harm later on in life.
SnapDragon
1mo ago
Isn't there a case to be made for an exception here? It's not some cheap "gotcha", there's an actual relevant point to be made when you fail to spot the AI paragraph without knowing you're being tested on it. The fact that @self_made_human did catch it is interesting data! To me, it's similar to when Scott would post "the the" (broken by line breaks) at random to see who could spot it.
SnapDragon
3mo ago
Uh, what do you mean we don't have self-driving cars? I took two driverless Waymo rides last week, navigating the nasty, twisting streets of SF. It drove just fine. Maybe you could argue it's not cost-effective yet, or that there are still regulatory hurdles, but I think what you meant is that the tech doesn't work. And that's clearly false.
Also, I'm a programmer and productively using ChatGPT at work, so I'd say the score so far is Magusoflight 0, my lying eyes 2.
Fantastic post, thanks! Lots of stuff in there that I can agree with, though I'm a lot more optimistic than you. Those 3 questions are well stated and help to clarify points of disagreement, but (as always) reality probably doesn't factor so cleanly.
I really think almost all the meat lies in Question 1. You're joking a little with the "line goes to infinity" argument, but I think almost everyone reasonable agrees that near-future AI will plateau somehow, but there's a world of difference in where it plateaus. If it goes to ASI (say, 10x smarter than a human or better), then fine, we can argue about questions 2 and 3 (though I know this is where doomers love spending their time). Admittedly, it IS kind of wild that this this a tech where we can seriously talk about singularity and extinction as potential outcomes with actual percentage probabilities. That certainly didn't happen with the cotton gin.
There's just so much space between "as important as the smartphone" -> "as important as the internet" (which I am pretty convinced is the baseline, given current AI capabilities) -> "as important as the industrial revolution" -> "transcending physical needs". I think there's a real motte/bailey in effect, where skeptics will say "current AIs suck and will never get good enough to replace even 10% of human intellectual labour" (bailey), but when challenged with data and benchmarks, will retreat to "AIs becoming gods is sci-fi nonsense" (motte). And I think you're mixing the two somewhat, talking about AIs just becoming Very Good in the same paragraph as superintelligences consuming galaxies.
I'm not even certain assigning percentages to predictions like this really makes much sense, but just based on my interactions with LLMs, my good understanding of the tech behind them, and my experience using them at work, here are my thoughts on what the world looks like in 2030:
- 2%: LLMs really turn out to be overhyped, attempts at getting useful work out of them have sputtered out, I have egg all over my face.
- 18%: ChatGPT o3 turns out to be roughly at the plateau of LLM intelligence. Open-Source has caught up, the models are all 1000x cheaper to use due to chip improvements, but hallucinations and lack of common sense are still a fundamental flaw in how the LLM algorithms work. LLMs are the next Google - humans can't imagine doing stuff without a better-than-Star-Trek encyclopedic assistant available to them at all times.
- 30%: LLMs plateau at roughly human-level reasoning and superhuman knowledge. A huge amount of work at companies is being done by LLMs (or whatever their descendant is called), but humans remain employed. The work the humans do is even more bullshit than the current status quo, but society is still structured around humans "pretending to work" and is slow to change. This is the result of "Nothing Ever Happens" colliding with a transformative technology. It really sucks for people who don't get the useless college credentials to get in the door to the useless jobs, though.
- 40%: LLMs are just better than humans. We're in the middle of a massive realignment of almost all industries; most companies have catastrophically downsized their white-collar jobs, and embodied robots/self-driving cars are doing a great deal of blue-collar work too. A historically unprecedented number of humans are unemployable, economically useless. UBI is the biggest political issue in the world. But at least entertainment will be insanely abundant, with Hollywood-level movies and AAA-level videogames being as easy to make as Royal Road novels are now.
- 9.5%: AI recursively accelerates AI research without hitting engineering bottlenecks (a la "AI 2027"), ASI is the new reality for us. The singularity is either here or visibly coming. Might be utopian, might be dystopian, but it's inescapable.
- 0.5%: Yudkowsky turns out to be right (mostly by accident, because LLMs resemble the AI in his writings about as closely as they resemble Asimov's robots). We're all dead.
SnapDragon
4mo ago
Er, but "man" and "woman" really do have an objective scientific meaning, unlike "relative", which is a social convention. (Note that it would be equally incorrect to say "an in-law is your blood relative".) So I don't agree with your analogies; saying "trans women are women" is just an incorrect statement of fact, rather than describing social conventions.
That said, I do think your framing of transness as a social status is reasonable. If we were simply allowed to say someone was "living as the other sex", rather than the Orwellian thought control that the ideologues insist on, I think it wouldn't be nearly as controversial.
Well, I don't think your analogy of the Turing Test to a test for general intelligence is a good one. The reason the Turing Test is so popular is that it's a nice, objective, pass-or-fail test. Which makes it easy to apply - even if it's understood that it isn't perfectly correlated with AGI. (If you take HAL and force it to output a modem sound after every sentence it speaks, it fails the Turing Test every time, but that has nothing to do with its intelligence.)
Unfortunately we just don't have any simple definition or test for "general intelligence". You can't just ask questions across a variety of fields and declare "not intelligent" as soon as it fails one (or else humans would fail as soon as you asked them to rotate an 8-dimensional object in their head). I do agree that a proper test requires that we dynamically change the questions (so you can't just fit the AI to the test). But I think that, unavoidably, the test is going to boil down to a wishy-washy preponderance-of-evidence kind of thing. Hence everyone has their own vague definition of what "AGI" means to them; honestly, I'm fine with saying we're not there yet, but I'm also fine arguing that ChatGPT already satisfies it.
There are plenty of dynamic, "general", never-before-seen questions you can ask where ChatGPT does just fine! I do it all the time. The cherrypicking I'm referring to is, for example, the "how many Rs in strawberry" question, which is easy for us and hard for LLMs because of how they see tokens (and, also, I think humans are better at subitizing than LLMs). The fact that LLMs often get this wrong is a mark against them, but it's not iron-clad "proof" that they're not generally intelligent. (The channel AI Explained has a "Simple Bench" that I also don't really consider a proper test of AGI, because it's full of questions that are easy if you have embodied experience as a human. LLMs obviously do not.)
In the movie Phenomenon, rapidly listing mammals from A-Z is considered a sign of extreme intelligence. I can't do it without serious thought. ChatGPT does it instantly. In Bizarro ChatGPT world, somebody could write a cherrypicked blog post about how I do not have general intelligence.
SnapDragon
3mo ago
I just can’t take these people seriously. They’re almost going out of their way to be easy for any real authoritarian government to round up, by being obvious about their identity.
LARPing is fun. They believe that they believe they're bravely resisting a dictatorship. But their actions make it clear that, at some level, they know there's no actual danger.
I consider it similar to climate activists who believe that they believe that the future of human civilization depends on cutting CO2 emissions to zero. And who also oppose nuclear power, because ick.
SnapDragon
2mo ago
As far as I can tell, the vast vast VAST majority of it is slop full of repurposed music and lyrics that get by on being offensive rather than clever. Rap artists aren't known for being intelligent, after all. I suspect most "celebrated" rap music would fail a double-blind test against some rando writing parodic lyrics and banging on an audio synthesizer for a few hours. Much like postmodern art, where the janitor can't tell it apart from trash.
There probably are some examples of the genre that I could learn to appreciate (Epic Rap Battles of History comes to mind), but it's hard to find them because of the pomo effect.
SnapDragon
2mo ago
Waymo has a lot of data, and claims a 60-80% reduction in accidents per mile for self-driving cars. You should take it with a grain of salt, of course, but I think there are people holding them to a decent reporting standard. The real point is that even being 5x safer might not be enough for the public. Same with having an AI parse regulations/laws...
SnapDragon
4mo ago
This is exactly @jeroboam's point - you say "AI is a junior engineer" as if that's some sort of insult, rather than unbelievably friggin' miraculous. In 2020, predicting "in 2025, AI will be able to code as well as a junior engineer" would have singled you out as a ridiculous sci-fi AI optimist. If we could only attach generators to the AI goalposts as they zoom into the distance, it would help pay for some of the training power costs... :)
It's weird and a surprise that current AI functions differently enough from us that it's gone superhuman in some ways and remains subhuman in others. We'd all thought that AGI would be unmistakable when it arrived, but the reality seems to be much fuzzier than that. Still, we're living in amazing times.
SnapDragon
4mo ago
Sorry, this is just tired philosobabble, which I have no patience for. All the biological ways to define man and woman agree in >99% of cases, and agree with what humans instinctively know, too. If you want to pretend that obvious things aren't obvious for the sake of your political goals, I'm not going to play along. That's anti-intelligence.
You know, sometimes pools just accidentally lose their exit. Common engineering mishap. My sincere condolences to those affected.
More options
Context Copy link