site banner
PaperclipPerfector  1mo ago (text post)   24110 thread views

Culture War Roundup for the week of May 18, 2026

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

5
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

AI bros still in shambles, news at 7.

A few weeks ago, Anthropic made a post about their new model, Mythos. As has been done by other members of the AI industry as far back as the release of GPT 2, the creators of it said it was too dangerous to release. The headline feature of Mythos, at least as described by Anthropic, was not code generation. Instead, they specifically hyped it as the most amazing thing ever for finding security vulnerabilities in code.

Several people, including here on this forum, shared the hype. As usual, I remained unconvinced. I've mentioned elsewhere that I don't think AIs are inherently incapable of finding security vulnerabilities in code, my main skepticism is that they will generate lots of false positives in the process that will make them a lot less useful than the companies selling them have advertised. And more importantly, I think they are currently incapable of designing and maintaining any significant projects that go beyond a basic bitch CRUD application or things of that sort. I'm also skeptical that there is all that much room for growth or improvement beyond their current capabilities, for a number of reasons that I won't get into right now.

But enough about my opinions, I'm just a retarded code monkey doing API integrations for boring tax software. Enter Daniel Stenberg, the creator and maintainer of curl. For those who don't know, if you have a program or library that makes HTTP requests, there is an extremely high likelihood that it is using curl under the hood. It's basically one of the foundational pieces of modern digital infrastructure, a "project some random person in Nebraska has been thanklessly maintaining since 2003", as XKCD might put it: https://xkcd.com/2347/

Stenberg/curl was one of the projects that was offered early access to Mythos. However despite being promised access initially, it took several weeks to get it. And even then he suddenly was no longer being offered direct access, but was offered to have someone else run Mythos against his codebase for him and to then share the results with him. This is a big red flag for me, because if Mythos does actually generate a lot of noise/false positives, it would make sense that Anthropic would want to hide that by running it themselves as many times as they could until it actually generated some real, actionable results.

In any case, the results that Stenberg got back were underwhelming. Mythos claimed to have identified 5 vulnerabilities. After investigating all of them, Stenberg and his team determined that only one of those was a vulnerability, and a low severity one at that. In Stenberg's own words: "curl is certainly getting better thanks to this report, but counted by the volume of issues found, all the previous AI tools we have used have resulted in larger bugfix amounts."

Most damning from Stenberg is this: "My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing."

So I'm asking @self_made_human and others who seem more on-board with the AI hype train: does this report from a knowledgeable and experienced developer change your opinions on the future trajectory of AI at all?

Full article by Stenberg can be found here:

https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-vulnerability/

Do you ever tire of this? I don't mean AI skepticism, I mean finding the least persuasive pretexts for it.

Enter Daniel Stenberg, the creator and maintainer of curl

from your own link:

Before this first Mythos report, we had already scanned curl with several different very capable AI powered tools (I mean in addition to running a number of “normal” static code analyzers all the time, using the pickiest compiler options and doing fuzzing on it for years etc). Primarily AISLE, Zeropath and OpenAI’s Codex Security have been used to scrutinize the code with AI. These tools and the analyses they have done have triggered somewhere between two and three hundred bugfixes merged in curl through-out the recent 8-10 months or so. A bunch of the findings these AI tools reported were confirmed vulnerabilities and have been published as CVEs. Probably a dozen or more.

I don't care to argue that Anthropic has greater AI than other players, I don't even believe this, by all accounts Mythos is pretty much Opus that's like 3x bigger. But Mythos finding anything on top of that is impressive enough, because presumably these tools have picked clean all curl vulnerabilities that are easy for AI to notice (on top of humans having hunted vulnerabilities in general for decades now). The real news for me here has been how tightly AI audits are already integrated into our core digital infrastructure.

If you want to deboonk AI hype, you've got to try harder. I propose attacking this.

I thought all the talk about software vulnerabilities would peter out for now, but I don't think that marketing is the only explanation.

Materialists are making the logically consistent assumption that if humans are computers, then AI is guaranteed to surpass our capabilities in every respect. So they predict a future which may not be real if materialism isn't real, and are hallucinating that such a future has arrived out of a cycle of fear and a desire to get ahead of it.

Strictly speaking, just because one hyped-up thing failed to hit the mark, it doesn't mean that it isn't coming, especially given the pace of developments. But Charlie Kirk said it right: AI is destined to throw our assumptions into chaos one way or another, and I, for one, am curious to see exactly what gets discredited as our knowledge and actual experience is forced to increase. Though it would be nice if we had a better understanding of things before we're forced to learn it inadvertently.

Materialists are making the logically consistent assumption that if humans are computers

Neuroscience still has a lot of ground to cover, but we already know the brain isn't a binary computer. It seems to me that one very easily could be a materialist and think that the brain is not a computer and I've always been a bit puzzled by the consistent tendency to equivocate them.

Neuroscience still has a lot of ground to cover, but we already know the brain isn't a binary computer. It seems to me that one very easily could be a materialist and think that the brain is not a computer and I've always been a bit puzzled by the consistent tendency to equivocate them.

The claim isn't that the brain is a "binary computer", it's that it's that however the brain works, it does not have computational capabilities that go beyond what is expressible by a Turing machine. So far we haven't been able to come up with a physical system of whatever sort that everyone agrees is able to come up with results that something like a digital computer can not even in principle. Roger Penrose does think that the human brain is one of those, and some mathematical insights humans can have are literally examples of super-Turing computation, but most everyone else thinks he's being a crank about this.

The claim isn't that the brain is a "binary computer", it's that it's that however the brain works, it does not have computational capabilities that go beyond what is expressible by a Turing machine.

Your link says

the computational theory of mind (CTM), also known as computationalism, is a family of views that hold that the human mind is an information processing system and that cognition and consciousness together are a form of computation.

It then goes on to explain that, arguably, "everything is computer."

Perhaps the human mind is a computer in the sense that everything is, but there doesn't seem to be good evidence that it is a computer in the sense that the metaphor is helpful to understanding the human mind. The human brain does not create representations of stimuli, store them, manipulate them, and retrieve them later upon demand according to a series of algorithmic rules.

Perhaps the human mind can't perform any mathematical calculations that cannot be performed by a Turing machine, but that doesn't mean that saying it is a computer is a helpful analogy. A digital tape recorder can record any song that a record can, but it's not helpful to call a record player a computer either - the mode of operation is different.

So far we haven't been able to come up with a physical system of whatever sort that everyone agrees is able to come up with results that something like a digital computer can not even in principle.

While I am sure that "not everyone agrees" my understanding is that it seems pretty clear that the universe, itself, is not simulable.

Perhaps the human mind is a computer in the sense that everything is, but there doesn't seem to be good evidence that it is a computer in the sense that the metaphor is helpful to understanding the human mind.

That's the thing. People didn't decide a priori that "everything is a computer". People just went looking for things that can't be mapped into computers all over nature and never found one.

Perhaps the human mind can't perform any mathematical calculations that cannot be performed by a Turing machine, but that doesn't mean that saying it is a computer is a helpful analogy.

This is pretty much what the debate comes down to though, remember the original argument was about whether we should expect AIs to surpass humans in everything humans can do. People keep trying to claim that humans have some magical domains of competence that will remain out of reach of AIs. For this to be an useful argument against claims of AI doom, it needs to cash out as the human mind doing some sort of work that shows up as output in the world, like a symphony or a beautiful masterpiece on a canvas. The theory of computation is very different from actual computer engineering, and the Aeon magazine writer seems to not understand this. It doesn't say anything about bytes, files, subroutines, operating systems, databases, images or buffers, just that there is some finite-length (but probably very long) lawful process that generates the speech or movement that shows that the thinking happened, and that the process could be translated to be run by a Turing machine.

While I am sure that "not everyone agrees" my understanding is that it seems pretty clear that the universe, itself, is not simulable.

I'm not a theoretical physicist but I'm pretty willing to bet that a physics paper that appeals to Gödel's incompleteness theorem for wide-ranging claims about the ultimate nature of reality will not end up receiving wide scientific agreement. The Gödel argument is basically the same thing Roger Penrose goes on about, and it goes back to John Lucas in 1959. It's had plenty of time to convince people and as far as I understand it by and large hasn't done that.

Apparently a previous reply was eaten, my sincere apologies if this ends up a double-post.

That's the thing. People didn't decide a priori that "everything is a computer". People just went looking for things that can't be mapped into computers all over nature and never found one.

The fact that "people" latch on to an easy metaphor does not necessarily indicate that the metaphor is good. The fact that the people most familiar with computers latch on to this metaphor also does not necessarily indicate that the metaphor is good.

remember the original argument was about whether we should expect AIs to surpass humans in everything humans can do.

This wasn't my claim, though.

The theory of computation is very different from actual computer engineering, and the Aeon magazine writer seems to not understand this.

The Aeon author did tackle the idea that the mind is an algorithm, which is, as I understand it, part of the theory of computation. We have good reasons to think the brain does not run on an algorithm; as the author of the piece I linked to points out, memory is extremely inexact, which is the opposite of what we would expect if the brain operated in an algorithmic manner.

But to take a step back, even if we wish to draw a distinction between "computer as hardware" and "computer as information processing device" the linguistic overlap invites us to confuse the two. And I don't think this is good; the analogy breaks down quickly in practice and invites us to forget the massive differences between the brain and electronic computers; it's true the brain uses electrical impulses but it also uses chemicals and is much slower than a computer. This metaphor, turned loose into the wild, has led to the popularization of what should be obviously implausible ideas, such as "mind uploading" or even that a computer could have emotions that we know in humans are substantially influenced by hormones.

In short, the idea that the mind is a computer is a sloppy one even if the motte is more defensible than the bailey by far precisely because the word "computer" makes it inherently a metaphor that yields a motte-and-bailey, even subconsciously.

The Gödel argument is basically the same thing Roger Penrose goes on about

I am not a theoretical physicist, or a mathematician, or a neurologist, but I am pretty sure you are wrong.

As I understand it, it works something like this. Gödel's incompleteness theorem says you can't algorithmically "solve" math (in the sense that there's not a super-algorithm that can do all mathematics). Penrose said "aha but humans can so we're BETTER THAN TURING MACHINES." The skepticism of Penrose isn't that Gödel is wrong, it's about whether or not humans can do that. If Gödel's incompleteness theorems suggest that our universe isn't a simulation, that's a different line of argument.

The Aeon author did tackle the idea that the mind is an algorithm, which is, as I understand it, part of the theory of computation.

Yep, this is a much less prone to confusion way of saying it than "the mind is a computer".

We have good reasons to think the brain does not run on an algorithm; as the author of the piece I linked to points out, memory is extremely inexact, which is the opposite of what we would expect if the brain operated in an algorithmic manner.

And this is utterly confused. Douglas Hofstadter's cartoon illustrated the error pithily way back in Gödel, Escher, Bach. The algorithm is exact (the small, correct sums in the Hofstadter cartoon), but it's also too precise and constrained to do mind-like stuff directly in the small. Instead, the mind runs on a sort of virtual machine (big numbers built from the small sums in the cartoon) built up by the algorithm that can do complex pattern recognition and creative solutions, but is also constantly getting things wrong. As we see from AIs, virtual machines like this can be implemented on silicon just fine and they exhibit the same behavior of being able to do difficult useful stuff but also constantly getting details wrong on their own.

In short, the idea that the mind is a computer is a sloppy one even if the motte is more defensible than the bailey by far precisely because the word "computer" makes it inherently a metaphor that yields a motte-and-bailey, even subconsciously.

I sorta agree here. It's basically an accident of history that "computers", things with hard drives, keyboards, operating systems, files, RAM and CPUs, and "computation", the evaluation of primitive recursive mathematical functions which matches what a Turing machine (which, again, isn't a "machine" that you build from wires and bolts, but a mathematical construct), ended up using the same terminology up to "computer" being right there in the name "computer science". This is why the cognitive science school is called "computationalism" instead of "computerism" and the practitioners optimistically thought that given a name like that, obviously people would think Turing machines, not quad core Mac Pros.

As I understand it, it works something like this. Gödel's incompleteness theorem says you can't algorithmically "solve" math (in the sense that there's not a super-algorithm that can do all mathematics). Penrose said "aha but humans can so we're BETTER THAN TURING MACHINES." The skepticism of Penrose isn't that Gödel is wrong, it's about whether or not humans can do that. If Gödel's incompleteness theorems suggest that our universe isn't a simulation, that's a different line of argument.

The problem with Penrose's argument is that humans are doing math pretty much as you'd expect if constrained by Gödel. By stumbling into theorems, working hard trying to prove them, and sometimes finding themselves stuck and unable to show something as either true or untrue. The crackpot smell with the physics paper is that Gödel's theorem is ultimately pretty limited. It says that any formal system powerful enough to do any sort of interesting math in allows stating the equivalent of the liar's paradox, which cannot logically resolve to be either true or false, therefore you can't have a mechanism for determining the truth of any proposition because you have liar's paradox propositions floating around. The equivalent impossibility theorem for computer science is the halting problem, you can't write a program that looks at the source code of any program and tells whether the program will terminate. For simulations, this would be saying something like that you need to actually run the simulation to see what kind of state it ultimately ends up in (and whether it stops at a steady state or goes on forever), and can't just look at the simulation's source code and figure it out. But it doesn't prohibit running the simulation and looking at what happens in it while it's running.

Even assuming the article is correct, I'm not sure it'll tell us anything useful about human capabilities versus silicon. Halting problem style arguments do claim that we can't build a literal machine-god that can figure out the exact trajectory of our universe ahead of time just by thinking hard. But that's not necessary to have machines that are better at doing everything humans value doing.

Instead, the mind runs on a sort of virtual machine (big numbers built from the small sums in the cartoon) built up by the algorithm that can do complex pattern recognition and creative solutions, but is also constantly getting things wrong.

This is a possible explanation, but as far as I can tell, not a necessary one, except inasmuch as one could stretch the word algorithm - which carries a connotation (or perhaps definition, if you cherry-pick one) of precision and repeatability - to encompass any process - although perhaps we are talking past each other here. Certainly the brain has deterministic aspects. But because it's a physical organ, it doesn't seem to behave algorithmically. Even if there is an underlying algorithm (and certainly I imagine there's an underlying process or, more properly, series of processes) it's so confounded by biological processes that I still have qualms about the word choice.

Even assuming the article is correct, I'm not sure it'll tell us anything useful about human capabilities versus silicon.

Yes, I think that's right. I brought it up because the universe is a physical system that can do things an algorithm can't.

Halting problem style arguments do claim that we can't build a literal machine-god that can figure out the exact trajectory of our universe ahead of time just by thinking hard. But that's not necessary to have machines that are better at doing everything humans value doing.

Yes, and I am much more irritated by the former sorts of arguments than the latter sorts of arguments.

My personal take is that AIs are likely to continue to be "spiky" in their intelligence for the near future but that's not because of abstract beliefs so much as it is just observing their overall trajectory and what I know about how they work. There will probably always be things that humans are better at doing, but I think that is a claim I can make with some confidence because humans like doing things like procreating, not because of Gödel's incompleteness theorem. Even if Penrose is right, it doesn't seem to me like it tells us much about the capabilities of silicon in most practical matters.

More comments

Materialists are making the logically consistent assumption that if humans are computers, then AI is guaranteed to surpass our capabilities in every respect. So they predict a future which may not be real if materialism isn't real, and are hallucinating that such a future has arrived out of a cycle of fear and a desire to get ahead of it.

I don't think this makes sense. You don't have to be a materialist to believe that AI is capable of surpassing human capabilities in all strategically relevant respects. It may very well be that only creatures with non-material souls can have qualia, but AI doesn't need qualia to destroy the world, and it certainly doesn't need qualia to wreck the economy in a mundane sense where it doesn't even go rogue.

You'd have to be not only a non-materialist, but someone who believes that the soul is doing a lot of the 'thinking' in a practical sense, for this to be otherwise - and I don't think that's a mainstream opinion even among dualists. But even then - even if you believe that a material machine can never replicate what happens in a human's mind when the human thinks about a problem, this is no guarantee that the AI can't arrive at a functional answer by different, possibly more efficient means.

I agree, except that if you start with the assumption that one doesn't yet know what the capabilities of AI are, then one rationally ought to keep space for skepticism of doomsday scenarios.

But you're right, and I don't assume that trouble isn't coming; I just saw the obvious other explanation for the talk of vulnerability-finding AI and determined based on how people were behaving that hype was the more likely explanation, this time. And I think that fear is primarily driven by the materialism of our times.

After all, when people talk about artificial intelligence replacing humans, the unstated premise is that humans are really just computers or not much better. See how easily they can do what humans can do? Haven't they passed the Turing Test?

Obviously, this is an attempt at mind reading, but I think it is a better explanation than marketing. As a marketing strategy, intentionally making promises that will obviously be falsified and talked about widely when the product is released seems silly.

So funny story. I've finally been pressed into using AI at work. I work on a closed network, but they run an LLM locally, so I basically use it the same way I use google these days, since all search engines have turned into LLMs. It's good enough when I have a quick question about syntax I've forgotten, or an API I can't access the documentation for. I still refuse on principle to have it write any code for me though.

Ah. I'm glad I'm not the only one who's come to the conclusion that millions(billions?) sunk into LLMs has basically just re-invented google search from 2015.

Sort of related…

I’m a latecomer to IT and information security. I spent most of my career in sales until going back to school for InfoSec. I started off in a help-desk role three years ago and I’m currently the information security engineer for a small IT team. I basically handle all security tasks: network, web, IAM, audits, etc. I’m 42, so this was a later-in-life transition. My boss is younger than me by at least 7 years and is far more knowledgeable than I.

Anyways…I have found myself to be very interested in application/web app security. The thing is, I can’t code. I rely mainly on vuln scanning and static code testing, with a little bit of pen testing knowledge thrown in. Any advice on where I should start if I want to learn more about app development and coding?

The thing is, I can’t code. I rely mainly on vuln scanning and static code testing, with a little bit of pen testing knowledge thrown in. Any advice on where I should start if I want to learn more about app development and coding?

Learn C to the point where you understand how to work with pointers and the whole business of a function receiving the pointer of a memory region and doing stuff to it. This is old-school, there's little new programming that should be done with C because of how hard it is to write secure programs in it. But it's great as a model that fits in your head for how the ground floor works in an actual computer program. Doubly so if you're interested in infosec, since a lot of attacks involve impendance mismatch between the conceptual idea of a program and the boots-on-ground reality of its runtime that's probably dealing with something written in C near the bottom.

You'll want to learn another programming language to write actual software in, but whatever you pick, if you know C, you now have the mental tool of asking "what kind of C program does this weird thing this programming language does reduce to?", which will hopefully help you see it as more of a useful tool than an inscrutable black box.

https://automatetheboringstuff.com/

My go to recommendation.

I seriously recommend either a good intro class or a good book for self study. (My recommendations on those would be pretty outdated now, so I can't offer any names myself.)

What WhiningCoil says about programming being a diverse set of skills in practice is true. But there is a core aptitude of thinking algorithmically. Some people can do it off the bat, some people can't do it at all, and some people need to try it from several angles before it clicks. This isn't really a matter of being smart enough; once you're over a certain threshold of intelligence, there just seem to be some people who are wired for it and some who aren't.

So I'd start with that. If it clicks, you can move on to study the other stuff in whatever way is best for you. If it doesn't, you can know that you gave it a fair shake.

Edit: As an addendum, I recommend learning your second programming language soon after your first. Some people fret about this and think it will be harder than it is. But it isn't that hard, and having experienced it will change how you evaluate your tooIs.

This is going to sound insane, but learn to write music in standard notation. I'm a lot of ways, it's a very simplified programming language.

Flashback to ABC notation

I donno man. I took to coding like a fish to water. And "coding" is really like, a half dozen skills put together. It's knowing the language you want to code in. It's knowing the ecosystem of libraries that probably do most of the work for you. It's having some knowledge, if imperfect, of what's probably going on under the hood with respect to threads, memory, disk access, garbage collection, etc. It's knowing how not to code yourself into a dead end, or unfuck yourself if you find yourself there.

It actually reminds me how woodworking isn't just cutting and assembly wood. It's making a design, picking out planks, milling to s4s, factoring in wood movement, sanding, finishing. The part most people think of as "woodworking" might actually be 5% of the task. It's the exciting part most youtube woodworkers focus on. But it's still probably the smallest part of the job.

And I went to school to learn how to code. I'm not sure I'd recommend that mid career. Maybe take some online classes. Open source can be intimidating, but I think contributing to it did more than anything to grow my skills and increase my confidence. Diving into a foreign, mature code base and learning how they do things is also a huge part of the job.

since all search engines have turned into LLMs

Google appears to have actually dropped their full Boolean search functionality, I assume because of this.

It's going to become a huge problem (or, at a minimum, extremely annoying), particularly in parts of my line of work.

Google appears to have actually dropped their full Boolean search functionality

Well, crap. I may now be finally forced to shift to a different search engine because of this, but they all seem to be rushing full tilt like the Gadarene swine into AI-ifcation.

My expression right now: 😠

I may have overstated the problem - I need to test it more, I was having problems with the exact search function and it seems Google has a "verbatim mode" that might assuage my concerns - but I definitely am not happy with the overall trajectory.

Verbatim and minus have just meant "more/less of this please" to google for years now -- well before LLM influence. I'm not sure why exactly, but corporate policy seems to be that (even setting aside sponsored results) the algo knows what you want better than you do. And the algo is getting worse.

Usually in the past if I copy/pasted something into Google in quote marks, it would quickly point me towards the right thing.

A week or two ago when I was working on a project that required this, I had a weird experience. If I'm recalling the exact sequence right, it told me it didn't have any matches - but then, when I scrolled down, the correct match was something like third from the top - the algo seemed to only be checking the preponderance of the words, and thus even when it could correctly source what I was looking for, it wouldn't flag as a 100% match.

So even though it had exactly what I was looking for, it didn't act as if it did.

Even when it does point you to the right thing, it is also showing you other things now -- in the deep(ish) past, if you put something in quotes it would only show results containing that string. Similarly (although I think this went away first), a search for -(thing you don't want to see) used to result in zero results containing that term -- now if you search for "used cars -chevy" it probably shows you fewer chevys than otherwise, but you are still going to see some. Particularly harmful when you are looking for something with one extremely common straightforward set of results (that you are not interested in) and an alternate niche interpretation. (the thing you want to find!)

AI influence seems to be making this a bit worse, I suspect since the "this is probably what he really wants" is more strongly weighted -- but it might be corpus frequency effects too I suppose.

What's frustrating is that I am pretty sure a nonzero portion of this is simply due to boost ad revenue.

Death by a thousand straws on the back of the goose that laid the golden egg.

I'm not sure why exactly, but corporate policy seems to be that (even setting aside sponsored results) the algo knows what you want better than you do. And the algo is getting worse.

The version of this that I hate the most right now, merely due to exposure, is in Windows, where the bottom-right notification pop-up gets selected or ignored if you click on the area just a few pixels out of it, as if I had accidentally clicked just outside the borders of it. No, I clicked on that specific pixel on purpose, because that pixel had the specific UI element that the pop-up box covered up that I wanted to select! If I click on a pixel directly adjacent to the pop-up box, I want it to be interpreted no differently from if I clicked on a pixel 500 away from the pop-up box. The only justification I can think of is for touchscreens, but those pop-up boxes aren't exactly tiny, and making UI behave differently based on input device (mouse vs touchscreen) is something that should be very very possible in Windows.

I'm showing my age perhaps, but I swear there was a time when double-clicking a word in windows selected just that word -- I understand that sometimes people would also want the trailing space, but now even if you drag-select, that gets helpfully added in many programs (eg. Word).

Clippy lives on as a sloppy ghost in the machine...

Re-endorsing Kagi, another search engine

Seconded! I was skeptical about paying for search, but it's so much better than Google these days. I pretty quickly was convinced it was worth the monthly subscription.

Thank you!

Apparently it's somewhat vulnerable to prompt injection as well.

https://www.usatoday.com/story/tech/2026/05/22/google-search-bar-ai-broken-disregard-ignore-dismiss/90219026007/

Darkly amusing to imagine LLMs putting me out of a job, not because they are better at what I do, but because Google for some reason decided to gouge out their own eye.

I guess their quality has been slipping for some time, but the other day it started giving me screwy results when I was hunting for specific phrases. I guess I will have to make sure that "verbatim mode" is switched on whenever I search for an exact phrase, now...and then hope they don't get rid of that, too.

I was trying to google whatever happened to that guy who ran down a Christmas parade. I remembered almost no details about it. Not the name, location, etc. Google's LLM was adamant that no black man had ever done anything like that, and explicitly said only white people had. It was only displaying search results about Charlottesville, and how the guy who did it got what was coming to him. I was trying to put together a rebuttal to a post last week or two on the Charlottesville Unite the Right incident. I think Google somehow knew that, because all the LLM summaries were preemptive rebuttals to the information I was attempting to find.

It made me highly skeptical of the narrative being pushed by the OP's "exhaustive" research. Especially when my own search attempts were so heavily guard railed to keep me on narrative.

I fucking hate this brave new world.

I did eventually find the information, and now for whatever reason it comes up readily. It was Darrell Brooks and he attacked a Waukesha Christmas Parade. He got the book thrown at him.

How could you ever forget Darrell Brooks? We have multiple Marseys of him over on rdrama.net

The whole trial was an absolute hoot and we even managed to get ourselves mentioned during the trial through one of our operations where we pretended to act like we'd put undue influence on the jury...

Google's LLM was adamant that no black man had ever done anything like that

No, you see, that's because if you remember the reporting at the time, it was the vehicle what done it, the evil machine. The car or truck took it into its head to just run out of the driver's control and charge into a parade all on its own initiative.

There was some mockery of the phrasing about this on social media, if you read the right websites. Brooks insisted on being his own representative at trial which led to some very entertaining moments.

The information shows up as the first search result. You say "For whatever reason" it shows up now, but what is your theory here- the Google AI somehow knew that you, specifically, were looking for wrongthink and tried to foil your attempts, but then elected not to do that for anyone else? I'd be curious to know exactly what your query was. You searched for something like "black guy who drove a car into a parade" and the AI summary posted text saying this never happened and only white people have ever driven cars into crowds? How very odd.

My guess is their LLM over indexes on the recent search history and what you click on. So likely WhiningCoil vaguely described the incident with perhaps incorrect info and with the low information query Google returned bad results. He clicked on them to see if they were the thing he was thinking of and the LLM got that irrelevant stuff stuck in its context. I have similar issues when using OpenAI models professionally and personally.

Searching "black guy who drove a car into a parade" returns the wiki article on the attack as the first result and has the same info in the AI box.

You should read until the end of my post.

I had a similar experience a little over a year ago

https://www.themotte.org/post/1231/smallscale-question-sunday-for-november-3/264751?context=8#context

I feel like a lot of people in these replies are talking past each other.

My 2 cents:

Are LLM's useful tooling for finding vulnerabilities for security researchers?

Yes, I think this is undeniable at this point; LLM's are exceptional at uncovering software flaws, bugs, and vulnerabilities, and are going to significantly change how cybersecurity is practiced, as can already been seen by how vulnerability disclosures have recently quantitatively spiked like crazy.

Is Mythos better than the other available models at finding and exploiting vulnerabilities?

Yes, Mythos really being a stronger model for cybersecurity applications is almost certainly the case: this XBOW report is a good read on its capabilities.

Is Mythos a super-hacker that's going to break cybersecurity for good?

No, this seems unlikely and driven by good marketing from Anthropic and online hype. Mythos isn't making the Move 37 for cybersecurity or discovering vulnerabilities beyond human comprehension, it's just an iterative improvement over the current tooling combined with a lot more compute and attention suddenly being used to uncover security vulnerabilities. I suspect that the same amount of compute, security researcher attention and buy-in for Project Glasswing applied to the previous generation of frontier models would have uncovered the majority of security issues that Mythos did.

It's also worth noting that there are apparently 11 Curl CVE's in the current release cycle, where the new CVE's did not use Mythos, which seems to disprove the idea that Mythos was not all that effective on Curl because it was uniquely hardened or secure.

Should LLM's being good at vulnerability discovery and theorem proving be an update on LLM's eventually reaching AGI?

YMMV, but to me, the recent headline mathematics and cybersecurity achievements haven't really changed my view that AGI emerging from LLM's seems unlikely. From an outsider's perspective, most of the recent gains in model performance look to have come from RLVR on coding, math and cyber. While very effective at improving performance on those tasks, it seems that RL has largely failed to further generalize intelligence beyond the specific RL'd areas, and if you look at SimpleBench or the AI RP community, seemed to have regressed performance in other areas of intelligence.

I think it's telling that all of the achievements of LLM's being held up over the past ~18 months (METR eval, CCC compiler, theorem proving, cybersecurity), while extremely powerful and which make me bullish on the utility of LLM's, are all tasks limited by requiring an external oracle for verification, and where there's no penalty for failing during intermediate steps. I personally think it's quite likely that LLM's eventually become superhuman at proving theorems and exploiting vulnerabilities given sufficient compute, but still cannot manage a restaurant, write an interesting book or autonomously maintain a software project.

You may be interested in Beren Millidge's take on Mythos (i.e. it's all RLVR):

https://www.beren.io/2026-04-11-Thoughts-On-Claude-Mythos/

Eppur si muove.

The problem with all these demos is that the level of capital involved is well beyond what it would take to simply contract some world-class humans to do their thing and misattribute their work to AI.

Like Terence Tao’s enthusiasm for AI seems, uh, kinda synthetic tbh. I’m like 90% sure he’s contracted to use these fancy models and try to get them to do something cool, then post about his experience, with the tacit understanding that further contract money is dependent on him not saying "Well, that was interesting, but basically a waste of time. Back to doing what I was going to do anyway." If you really want to put on the tinfoil hat, he was placed in a precarious financial situation to help motivate him, which was definitely not coordinated or planned by the people who coordinate and plan everything. If there’s one thing the Trump administration would never do, its leverage state policy to manipulate markets.

I think that critique was reasonable even a month ago: most of the novel proofs discovered by LLMs could have been done by a modal grad student in the field, given time and motivation. Still useful, but picking off only mildly interesting results that haven't received much focus isn't world changing.

This particular (dis)proof, however, is quite different. It has received extensive attention. Research Problems in Discrete Geometry called it "possibly the best known (and simplest to explain) problem in combinatorial geometry." Surveys have been written on it. Erdos himself returned to it many times and tried your approach, offering a bounty for solutions.

If some billionaire had dedicated billions of dollars for a resolution of the conjecture, it seems quite possible that nothing would have come of it. Thomas Bloom in the companion remarks has some interesting speculations as to why it resisted human attempts for so long that are relevant, and the other remarks are interesting as well.

Still useful, but picking off only mildly interesting results that haven't received much focus isn't world changing.

It is. Quantity has quality of its own. Even if LLM peak below humans, a stupider brain that is inexhaustible, can work 24/7 and can be scaled to infinity means that a lot of intellectual things could be bruteforced in the million monkeys with million typewriters way. Throw at a million small problems and there will be breakthrough somewhere.

Also check this https://modelrift.com/blog/openscad-llm-benchmark/

What was magic 6 months ago is boring and insufficient now. Also couple of AI uses last week - to decrypt pragrmata save, find and edit values of the upgrade currencies, rehash and resign. Fix some blutooth issues - it took it half an hour but managed to pair the troublesome adapter and mouse combo.

Even if the technology stop dead in the tracks now - we will need at least five years until all the effects and possibilities are clear.

I'm not too surprised that a secure piece of software exists, or that it's only 6 MB zipped with more installations than there are humans on Earth and a 30-year history.

Why are you highlighting this anecdote so much?

a secure piece of software

I think this is an unlikely claim. curl helpfully lists past vulnerabilities. (Fun fact: they stopped awarding bounties for vulnerabilities when people began posting AI slop bug reports, wasting their time.)

I do not think that "curl does not have any more medium-or-high level exploits beyond CVE-2026-7009 and CVE-2026-7168, so even an ASI could not have found any" is true.

Don't get me wrong, I think curl is certainly in the rightmost percentiles of software security (alongside openbsd), and an interfacing library (i.e., tons of attack surface) with a whopping 176kLOC and only 188 CVEs so far (despite heavy auditing) is pretty amazing, even more as it is written in C. It is entirely possible that Mythos will turn less-audited codebases (e.g. closed source or more niche open source) into a blood bath.

But I still think Stenberg's (not entirely dismissive) take is a good one to update on. Much of the software industry is very much on the AI hype train, and for the AI companies hype seems to be the main product. I would not expect Microsoft to come forward and call Mythos not a big deal (unless they are hyping up ChatGPT, of course), for example.

It is entirely possible that Mythos will turn less-audited codebases (e.g. closed source or more niche open source) into a blood bath.

But so will almost any other capable model.

Yes, or a bright teenager with nothing better to do, for that matter.

(Though there are certainly orders of magnitude more people with an LLM subscription than people with the skills and diligence to find exploits the old-fashioned way.)

Yes, or a bright teenager with nothing better to do, for that matter.

The last time that was true was when ROP-s were in vogue.

Most of the incapable models too, from what I've seen of internal systems at client sites.

I think the expectations of AI believers and the hype pushed out by the company is so absurd that it's quite easy to be considered an "AI skeptic" even if you're relatively bullish on AI. Like even if I were to believe an AI god were to come soon, we're just not getting 10% growth year on year. Not happening. Regulatory barriers alone make it impossible, and then there's diffusion problems, and then there's the fact that we just can't build up enough energy to scale growth that fast even if there were 0 regulatory barriers.

I do think the fact that the real world results never match the measured increases in AI capabilities is kind of indicative of the problem here. It's very easy to train AI in kind environments with clear feedback loops, but wicked environment outcomes are all over the place.

On one hand.

On the other hand:

Did you ever get around to trying my suggestion for setting up a code harness and predigesting your code base?

So I'm asking @self_made_human and others who seem more on-board with the AI hype train: does this report from a knowledgeable and experienced developer change your opinions on the future trajectory of AI at all?

I don't really have any strong opinions on what one dude has to say about about a model I can't otherwise evaluate myself, but in your own article the guy you're apparently claiming is skewering ai and that should put us in shambles also said:

We also see a high volume of high quality security reports flooding in: security researchers now use AI extensively and effectively.

Like, I dunno, man. Do you not feel like the goal posts are shifting here? It's useful but one report from mythos on one repo where the guy said he was disappointed that there weren't more bugs found because other AI tools had found more(Which were already patched and thus not available for mythos to find)? This is your justification for the whole of AI being slop and hype?

I guess I update slightly in favor of mythos being closer to the current public sota rather than a league ahead of it. Perhaps the Curl codebase is just actually so tight that the whatever IQ equivalent level security expert that mythos represents wasn't about to find much, I promise you that other projects are not so tight.

curl is one of the most fuzzed and audited C codebases in existence (OSS-Fuzz, Coverity, CodeQL, multiple paid audits). Finding anything in the hot paths (HTTP/1, TLS, URL parsing core) is unlikely.

Do you not feel like the goal posts are shifting here?

The original claim Anthropic made was that Mythos could do all of this independently. That it didn't need a highly experienced security researcher guiding it. In fact that the reason it was so dangerous is because any lay-person could use Mythos to "hack the planet" It's not goal post shifting to point out, no Mythos is just a SOTA tool and like most SOTA AI tools it works better by having an experienced human guiding it on what to code, look for, design the system etc. The AI ecosystem is very hype oriented, people claim far more than what is realistically delivered.

The amount of steering it needs is still totally speculative. There are a thousand reasons you'd run the queries yourself rather than give some rando model access. And @ChickenOverlord goes well beyond claiming the tools are moderately over hyped, he's argued extensively that they're basically useless for coding.

To be clear I am not an AI bear of @ChickenOverlord's persuasion, I am an AI bull, but very cynical on the marketing hype. I think his opinion that they are useless for coding does not align with my opinion. But I do think the original goalpost that Mythos is not some super intelligence, first step to the singularity, type model has not been moved. Mythos is a SOTA tool like other SOTA tools.

The amount of steering it needs is still totally speculative.

Absolutely, but reasoning by abduction points it to being worse than the marketing would suggest.

Sure, if you want to quibble over how far of an advancement mythos was then I think there are a variety of reasonable opinions. But this is very much part of chicken's "Ai bulls in shambles" series of posts and I can't help but point out that the people he's calling out as being in shambles here were basically right in previous iterations if you take the supposed doomsayer's opinion at face value.

Over the past few weeks we've had several serious vulnerabilities found in the Linux kernel (CopyFail, DirtyFrag, PinTheft), and LLM assistance has reduced the gap between "suspicious bugfix smells like it might patch a vulnerability", "someone other than the reporter/reportee has PoC and/or a working exploit", and "attackers are deploying it live in the wild" to nearly zero time.

Curl is an unusually disciplined project, and I think it is hard to generalize lessons from it.

So I'm asking @self_made_human and others who seem more on-board with the AI hype train

Choo choo!

So it only found 1 minor vulnerability in curl that hasn't been fixed before (including by these high level human programmers)... but it did find a bunch of other vulnerabilities in other software? It is indeed still markedly stronger than its predecessors?

So the future trajectory is just the same as the current trajectory, the lines on the chart go up and everything the lines correspond to in the real world also goes up, albeit in a messier way.

If you're an AI skeptic, then I recommend to simply short Nvidia, Coreweave, cloud providers, HBM manufacturers like Micron... What does it matter how random people on the internet think, compared to making money? I put my money where my mouth is and bought AI stocks and made lots of money. Let money flow to those who are right. If you think you know better than Google, Amazon, Microsoft, Facebook and everyone else pouring money into AI hand over fist, then don't just say so, position yourself to exploit your superior insights.

If you're an AI skeptic, then I recommend to simply short Nvidia, Coreweave, cloud providers, HBM manufacturers like Micron

Thinking that a loss-leading strategy is not going to pay off for the current AI ecosystem and AI skepticism are not the same thing, is it? You can think that the AI is very impressive and also that there's no way that Anthropic will ever climb out of its hole, or alternatively you can be the fiercest AI skeptic in the world and think that everyone will pay billions for a glorified chatbot.

I think they are currently incapable of designing and maintaining any significant projects that go beyond a basic bitch CRUD application or things of that sort. I'm also skeptical that there is all that much room for growth or improvement beyond their current capabilities

That's what he thinks. Surely he should just put his money where his mouth is? If Anthropic AIs cannot design or maintain any significant projects beyond a CRUD application and this isn't going to significantly change then presumably Anthropic is not worth near a trillion dollars and so the biggest industrial buildout in human history is a waste of money.

The premise that they're incapable of doing anything beyond CRUD and yet also they're completing long expert-level cyber infiltration exercises is bizarre and incoherent to me... but that's what he thinks.

  1. Anthropic isn’t yet public. You can’t easily directly bet against it.

  2. To short Nvidia would require the belief the hyperscaler will abandon the scaling. That’s really hard to time.

  3. Are you shorting a bunch of companies that would be killed by AI? P/E for many suggest they are highly overvalued if AI can displace in the next year or so.

  4. It isn’t obvious hyper scalers are making rational decisions. Apple isn’t hyper scaling (it is leasing). Doesn’t seem like a terrible idea…

Yeah but why aren't the hyperscalers abandoning scaling? Microsoft, Amazon, Google, Facebook made a deliberate choice to halt buybacks and spend hundreds of billions on AI. They made this choice based on something, they're spending $700 billion this year! You don't invest that much as a modern financialized American corporation without being sure about what you're doing.

He should be thinking that, if further significant improvements are impossible, then capex will plunge as soon as this is realized. But this isn't happening, we see continual improvements on a monthly basis.

Apple is more of a hardware company, they have a different business model to Microsoft and the others. AI is understandably not their great strength. They might reasonably calculate that they are not going to win a struggle with Google Deepmind on AI with regard to talent or compute or determination. AI is the lifeblood of Google, devices are the lifeblood of Apple.

You do realize sometimes corporations make the wrong choice? Also corporations choose what the market rewards them for. If they cut capex because they didn’t think there would be major improvements, their stock would plummet because it would mean all of their prior capex spending will never make an ROI.

The market doesn't necessarily reward companies for investing, it rewards stock buybacks (which were all the rage amongst big tech up until the AI boom).

If they wanted to juice their stocks, they'd just continue buybacks rather than buying GPUs.

It'd be surprising if these large, old, well-established software companies all catch AI fever at the same time. These are all survivors of the dotcom bubble, not fledgling newcomers with more credulous leadership.

This just isn’t true. The market generally prefers buybacks to dividends due to EPS, etc. However, the theory behind distributions (including buybacks) is that a SH can generate more return with the cash compared to the company. If the company can generate a higher return with the cash, they would not distribute and the SH would enjoy value creation via higher stock price.

By forgoing buybacks and instead spending a bunch on capex, these companies are signaling they can make more on the cash compared to the general market conditions. This is certainly the story these companies are selling as well.

None of this means the companies are wrong. But right now they are being heavily rewarded for investing in AI. If they stopped and started doing buybacks, they’d almost certainly drop in value.

Finally established companies fail literally all of the time.

More comments

Taking a long position like you are is not comparable to your suggestion of a short. Shorts have uncapped risks and a much more specific time horizon. "The market can remain irrational longer than you can remain solvent" is much more true for shorts. There were people who shorted tech stocks in the 2001 bubble and went bankrupt because they missed the crash by few months. There were similar people during the housing crisis. That doesn't mean they were directionally wrong - it means that timing is hard.

Timing sure is hard. I managed to buy Micron at the top and so lost out there, it then recovered but it took a while.

Nevertheless, you can make money shorting if you're actually right. If you know things that others don't know, you can use this to your advantage. Don't blow your whole load in one year, keep some powder left for if the ponzi goes higher. There are ways to position yourself to profit from this, if the thesis is true.

It might be worth moving this over into the finance thread, but I am at least partially putting my money where my mouth is.

I'm of the opinion that the current LLM hype is starting to hit the second knee of the S-curve, both financially and technically.

Technically, exponential growth leads to exponential friction, and it looks to me like the real-world improvements in model capabilities are slowing down between generations. Anecdotally, it feels like the models are increasingly fungible and most of the ostensible improvements have come from harnesses, which are regular old software engineering. I think there's something there, but I think LLM tech represents a local maximum. I'm eagerly watching whatever Yann Lecun is cooking up at AMIL, because the general concept of a world model seems to map better to what we think of as "intelligence". His paper on energy models, specifically, is fascinating.

Financially, I think a lot about the "during a gold rush, sell shovels" aphorism. I also think about Buffett and Munger's rules of investing. Meta and Oracle are buying shovels, but using them to dig their own graves, so far as I can tell. I don't think Anthropic and OpenAI are ever going to be able to support their valuations, and per their S-1, xAI has already pretty much given up. If Google dies, it'll be for reasons other than AI spending. Nvidia has a good product and a good moat for now, but various specialized competitors are nipping at their heels while Chinese cards may develop into direct competitors.

In other words, I think the tech is going to continue developing, but I think a lot of the current players are in for a rude reminder of market on market fundamentals by 2H 2027 or so. I know you've bemoaned "financialization" in the past, but at the end or the day the economy is just people buying and selling things, and fuck me if it doesn't seem like some of these companies are trying to act like that's not true.

Where does that leave me? I'm moving down the stack. If the tech is going somewhere, it has to run on real things and interact with the real world eventually. Companies doing physical things are riskier to start than pure software, but they're less likely to get disrupted once they establish themselves. I've largely stopped investing in funds that hold significant amounts of meta, oracle, Tesla (because I think they may absorb SpaceX), and even Nvidia. On the other hand, I'm expanding my positions in funds that hold TSMC, ASML, and Lam Research. I am watching Cerebras, but I won't invest until I better understand how they're using software to get around defects on their enormous chips.

Fair enough, I guess that's a reasonable stance.

It's just that just today I see people online talking about Qwen 3.7 Max:

Over 35 continuous hours, Qwen3.7-Max executed 1,158 tool calls and 432 self-evaluations. It wrote, compiled, profile-tested, and repeatedly rewrote a production-grade SGLang Triton attention kernel. The resulting custom kernel achieved a 10x speedup over the official reference code. Engineers on forums noted that its ability to identify optimization bottlenecks after 30 hours of continuous operations represents "true industrial-grade autonomous engineering" rather than standard code completion.

Are they lying? Was the kernel made up? Maybe Alibaba is massaging the figures to some extent with the exact meaning of what a 10x speedup means in this context, dramatic speedups for just a few tasks being averaged out. Yet we know that other AI models can also do this kind of task, the general idea can't be just a lie. If it's not a lie, then surely this seems like a highly desirable, powerful technology that can substitute for high-end human talent to some extent. GPT5.5's verified mathematical conjectures seem hard to cheat. Kernels and mathematics seem to have real world value, as does whatever Anthropic's been doing with the war in Iran in terms of intelligence, rapid realtime assessment. Hard to get more real-world or frictional than warfare...

Are they lying? Was the kernel made up?

Cases like this, and the erdos problems, are exactly where LLMs shine. Problems with clear and unambiguous reward functions that are difficult to hack are perfect use cases. In the Alibaba case, they likely have an extensive set of characterization tests that guarantee consistent behavior. An LLM with a good harness can pound its head against those tests forever while simultaneously measuring the performance as a success metric. It will never get tired and it won't get sick of doing that kind of work.

There's definitely value there, but I don't know how much value. The combination of technical depth and strong guardrails make for a very schizophrenic kind of difficulty. Doing that kind of work is traditionally either the domain of a plucky junior with too much energy, or an insane wizard who claimed a broom closet as his office.

When we've experimented with that kind of optimization work at my employer, it tends to be very expensive, since most of the results come from the absolute tirelessness of the agent. In comparison, how much are you paying your junior? How much are you paying your wizard, and what is he doing if he's not doing that task? Security scans are a similar thing. Line audits aren't hard, but they're hella time consuming. As model costs rise (and they are rising per task completed when you compare any single vendor over time), it might legitimately be cheaper to throw interns at the problem than LLMs.

At least on the software side, I think there's a reasonable chance that what we're seeing is a temporary pop due to a lot of highly verifiable technical debt deadwood finally getting burned out, and that might not be a constant source of demand.

On the war side, I wish I knew more. The sensitive nature of the topic means that all parties are incentivized to obfuscate and dissemble as much as possible. It might legitimately be an ideal case. LLMs do well when you can accept 95% accuracy, and in something like intelligence analysis, 95% accuracy probably has the spooks all but shitting their pants.

Surely if you think AI is capping out then you should expect ASICS to be the play. QCOM and the like. I have a hedge in some of those in case scaling doesn't continue as I expect.

They're firmly on my "investigate" list.

I'm of the opinion that the current LLM hype is starting to hit the second knee of the S-curve, both financially and technically.

Astral Codex Ten on this exact topic

For the record, while I appreciate the name-drop, I've largely checked out of this debate. I read the article when it crossed HN, which I browse daily. The strongest critique of Mythos is that GPT 5.5 Pro reaches similar benchmarks while being cheaper and generally available. Which is to say: Mythos isn't quite as special as Anthropic would like, because a competing frontier model already demonstrates equivalent capabilities. See the problem there? Or, from my vantage point, the absence of one?

Why so checked out?

Not because I've recanted, and not because I've stopped believing my own forecasts. It's that anyone who hasn't gotten the memo by now is beyond my ability to help. I've been on this beat for years, sounding the alarm for about as long. Litigating whether each fresh data point lands above or below the trendline has stopped feeling like a useful expenditure of my evenings. I still have the arguments cocked and loaded, still bookmark whatever catches my eye, with roughly the clinical curiosity of an ICU physician watching creatinine and urea climb and eGFR slide in a patient with end-stage renal disease. Erdos problems falling like dominos and Terence Tau watching from the sidelines, Tim Gowers writing up breakthroughs from OpenAI's unreleased general-purpose models, METR's task-horizon metrics snapping like a mediocre school psychologist trying to score Einstein on the Stanford-Binet. (At some point the instrument stops measuring the subject and starts measuring its own inadequacy.)*

TL;DR: my supply of fucks is running thin. If you're pinging me hoping to extract an argument about AI capabilities, calibrate accordingly. I've got bigger fish to fry before I get thrown into the fryer myself. Good luck to whoever still has the energy for it.

*Go ask ChatGPT for citations and actual links.

METR's task-horizon metrics snapping like a mediocre school psychologist trying to score Einstein on the Stanford-Binet.

Einstein's IQ was probably about 140, so no. You don't understand what you're talking about here. Nice try though.

Sure buddy. The psychiatry resident who reads up on psychometrics for fun and is fully aware of the unreliability of standard IQ testing when we're going several sigmas away from the median of the distribution wouldn't have any idea about what he's talking about. Especially when his actual point is that trying to IQ test someone as far out of distribution as Einstein (famous for being the dimmest bulb in the shed)* is going to give unreliable results?

You wanna try telling me Feynmann had an IQ of 125? I'll believe you, or at least humor you.

Aight. Gotta hand it to you. Never had a chance of winning this argument. I concede.

*He's famous for only emitting a singular photon.

Especially when his actual point is that trying to IQ test someone as far out of distribution as Einstein (famous for being the dimmest bulb in the shed)* is going to give unreliable results?

Einstein was not out of the distribution. He was very smart, but not in a reality-shattering way, and he was very focused on his craft, like all successful smart people.

The psychiatry resident who reads up on psychometrics for fun and is fully aware of the unreliability of standard IQ testing when we're going several sigmas away from the median of the distribution wouldn't have any idea about what he's talking about.

I have a PhD in statistics. With all due respect, I know what physicians study, and while many of you are great healthcare practitioners, you do not study the quantitative.

Einstein was not out of the distribution. He was very smart, but not in a reality-shattering way, and he was very focused on his craft, like all successful smart people.

Einstein. The gentleman responsible for Special Relativity, General Relativity, the photoelectric effect (which actually got him the Nobel), Brownian motion as proof of atoms, mass-energy equivalence, and Bose-Einstein statistics. "Very smart, but not in a reality-shattering way."

I'm going to need a minute to process that. Possibly three. Fortunately my psychiatry experience prepares me well; I can usually recover from being utterly flabbergasted in 5 seconds or bust.

If two complete overhauls of how humanity understands space, time, gravity, and matter doesn't clear your bar for "reality-shattering," I'd genuinely love to know what does. Should he have collapsed the lightcone via propagating false vacuum decay? Manually torn the curvature tensor out of the universe and presented it to Bohr in a jar? What are you on about? What are you smoking?

As for “Einstein was probably about 140”: probably according to what? A preserved Wechsler protocol from 1905? A Stanford-Binet administered by divine revelation? Some conversion table from “invented general relativity” to “moderately gifted but not too spooky”? I am genuinely curious how you got to “Einstein probably had IQ 140”. I presume you've heard of something called a ceiling effect?

"Focused on his craft, like all successful smart people" really makes me wonder which Einstein you mean. The one who played violin semi-seriously, wrote political and philosophical essays by the bushel, corresponded with Freud about the psychology of war, lobbied Roosevelt about the bomb, and turned down the presidency of Israel? Monomaniacal indeed. The phrase "successful smart people" is also wonderfully convenient as a construction, since any polymath counter-example presumably just gets retroactively reclassified as unsuccessful.

I have a PhD in statistics. With all due respect, I know what physicians study, and while many of you are great healthcare practitioners, you do not study the quantitative.

With whatever respect you're due, and without further comment on the magnitude of that debt: British psychiatrists are held to higher standards than that. I'm held to higher standards than that, mostly by myself. I know the difference between Cohen's d and Hedges' g. My interest in entering a d-measuring contest with you is, by consensus values, small. It is roughly equivalent to my interest in arguing with you about the psychometric validity of the other form of g.

Don't believe me? Here's the MRCPsych Paper B critical appraisal syllabus.

I gave it last week. The headache is still bad enough that I'm not going to dig through my own post history to surface the times I've gone several layers deep into statistics arguments on this site. You're welcome to spend your time doing so, I value mine.

Lumping me in with the median doctor who thinks p<0.05 gud? Nice try though.

I'm going to need a minute to process that. Possibly three. Fortunately my psychiatry experience prepares me well; I can usually recover from being utterly flabbergasted in 5 seconds or bust.

You appear to attribute all outward intellectual achievement to latent g. But the correlation between g and such achievement isn't 1. A really high estimate of that correlation would be 0.70, so was Einstein a 4 SD physicist of his day, his expected g would be 2.8 SDs, which is 142. 142 is extremely intelligent and a focused person at that level can achieve great things. It's been found that the correlation between intelligence and chess is as low as 0.24.

"Focused on his craft, like all successful smart people" really makes me wonder which Einstein you mean. The one who played violin semi-seriously, wrote political and philosophical essays by the bushel, corresponded with Freud about the psychology of war, lobbied Roosevelt about the bomb, and turned down the presidency of Israel? Monomaniacal indeed.

He lived a long time and did physics as a profession, but yes he also had hobbies. This is typical of people in the 140s IQ.

With whatever respect you're due, and without further comment on the magnitude of that debt: British psychiatrists are held to higher standards than that. I'm held to higher standards than that, mostly by myself. I know the difference between Cohen's d and Hedges' g.

I'm glad you pay more attention to statistics than most psychiatrists. Maybe one day you can help replace the DSM with a quantitative approach. Have you read Eysenck on general psychoticism? It's in his book on genius.

My tips for you on intuiting why you probably over-identify people being over 150 IQ:

  1. high IQ is exponentially-increasingly rare because the bell curve is thin-tailed. Shrinkage is optimal; you would have less error if you shrink everyone's intelligence towards the median by 10% or so.

  2. Flashy outcomes are a result of intelligence as well as other factors like work ethic and fortune. Many people think a famous businessman, scientist, or writer has an outrageously high IQ, because they think their preferred intellectual status markers correlate at >= 1 with intelligence. They don't.

  3. The CEF (conditional expectation function) of g on outcomes might not be linear. The IQ wealth CEF is probably concave for instance. Nonetheless, lots of people believe that Elon Musk must be a genius, but his real intelligence is probably top 1% or 2%. He's exogenously fortunate and an outlier in other ways, like commitment to business, and these explain his wealth jointly with his intelligence.

Wasn't it a big thing though in statistics that often a phenomenon is nicely described by the normal distribution in the fat part, but has much more outliers than the exponential shrinkage of the bell curve tail predicts, and so there are all sorts of modified distributions with wider tails. Is it well known that intelligence isn't one of those?

  1. Appearances can be deceptive. I do not think that g is some kind of stamp that gets impressed on your forehead and then dictates the rest of your life without further environmental modification. If you wish to argue that certain factors/metrics only explain a limited fraction of the observed variance in outcomes, you should have lead with being more charitable and assuming that I know what I'm talking about. Doctors are not known, as a class, to be particularly stupid. Your assumption was and does remain incorrect.
  2. I appreciate the more substantial attempt at engaging with my arguments, and while I have genuine disagreements, I wasn't kidding about the headache. That strikes me as a remarkable approach to estimating Einstein's IQ, going from what it might have been, in theory, to what it might have been on a hypothetical IQ test he never gave. I recall that there are plenty of confounders for the chess and IQ stuff, probably Berkson's paradox, but I do not have the time to check. The rest of your arguments are tangential to any point I came in with the intention of litigating. I told you so. Now, if you had lead with these points, any semblance of rigor, or at least charity, I would engage more productively. Right now, I simply can't even if I want to.

If we define g factor as smartness then I can understand @dailydogma argument. His point basically seems to be, whatever causes Einstein's success in physics was not his g factor. There are lot of people who are very talented in a skill such that the second place in the world doesn't come close to the first place in ability. For them g factor may not explain the variance in ability but some other (maybe inborn and genetic) trait may explain that variance better, a trait which does not generalize well to other domains.

Nigel Richards is incredibly dominant in his field, he is the french scrabble champion. He is better than any computer at the game and he doesn't understand french at all. He only learned it for the game in just nine weeks. This is not an ability any other competitor shows. [https://youtube.com/watch?v=T-8NrvVqbT4] If you try to explain his success with g you would end up with a very simple question

"If his ability is so much greater than his opponents and this is primarily explained by his g then his g must also be far far greater than his opponents, why is he is not an equally prominent physicist or the like. G factor generalizes, he should be better at everything." A explanation for this can be he has some trait which doesn't generalize well but helps with his specific domain, so he doesn't have very high IQ (or g or intelligence) but he is still very good in his field.

This can also lead to a very smart but not an world shattering level smart Einstien who nonetheless can make the discoveries he did. Just look at latest models from OpenAI and how jagged they are, they can do Erdos problems but we are not at AGI are we?

Maybe Von Neumann was consistently a genius in may different fields and hence was smarter than Einstien.

If at the very top something other than g factors starts explaining most of the variance then you can easily have 130 IQ Feynman.

Edit: Math is very g loaded but once you filter for people with high IQ (120+ or 130+) the correlation would drop, and you would probably end up with very jagged talent rather than a generalized talent which is used in g. After filtering at 145 IQ threshold different cognitive abilities merely have 0.1 correlation https://doi.org/10.1016/j.intell.2017.07.004

I'm going to need a minute to process that.

I heard you can do that in just 5 seconds if you just run really really fast. Like, really fast.

And shrink when viewed by an external observer? I'm a grower, not a shower.

Can you post to some of your forecasts? I know that you are bullish on the tech - but my vague impression was that they survived a sanity check - aka were not like the wet nightmares of AI2027

My current timelines (stable for the last year or two) are 50% odds of AGI by 2030, 70% by 2033.

My operational definition of AGI is "can do ~everything a human can with a computer as well or better than the the median human", ideally a 130 IQ human. That focuses on real world tasks, and also considers speed and reliability. I consider ASI achieved when the models reliably beat the smartest humans alive at similar or lower figures for $/unit of cognitive output.

In other words, if you attach my version of AGI to a computer with access to the internet it can do anything a human could with the same affordances, about as well. Probably with a video feed and a virtual keyboard or mouse, but that's not a big deal. Current models are too spiky in terms of capabilities to count, particularly when it comes to agentic workflows like simply using vision and direct input to get tasks done. I can't solve an Erdos problem even if you give me 5 years to prepare, but I can do more with my desktop PC than Claude can, at least much faster.

I expect that the temporal delta from that version of AGI to true ASI is going to be rather short. Maybe a year or two, medium confidence guess.

So pretty mild stuff, although I do find your definition of AGI/ASI somewhat texas sharpshooter style. On the other hand no one seems to have to be able to define those things in a way that is better so there's that.

It's interesting times when I'm told that my forecasting of a 50% chance of AI becoming human-parity or better in 4 years is described as a tame take. Not complaining, just observing things with grim resignation. I'll know AGI is here when I see it, or a few years later, if unemployed.

I wish I'm wrong, and that I had been wrong so far. It's no fun engaging in arguments where you want your opponents to win.

The whole of civilization and industrialization has been underpinned by using energy to achieve superhuman feats. Most of the worries so far about AI are thin veil for some people's fear that their greatest labor asset will lose value rapidly. And probably some narcissistic injury if we are using LP definitions.

If I didn't get worried that we created cars that could run 20 times faster than a human 24/7 or that we have trivialized the magic of flying to the point that it is utterly trivial, boring and so on, why should I worry that some machine will think better than me. Hell - I should be worried if we don't invent such a machine. Whatever you can think of - we are running out of it - out of soil, out of biomass, out of oil. We need faster science progression to get out of the trap that is our lovely blue planet.

And your predictions are tame because they fit linear advancement at current rates. And we will probably get there even on log. Your definition of AGI is modest.

I think that Mythos like everything else from Anthropic is crap mixed with bullshit, but two things of note here - curl is absolutely tiny and is one of the most heavily scrutinized pieces of software ever written. The other is probably putty. And the easy hanging fruit was already picked.

This is a big red flag for me, because if Mythos does actually generate a lot of noise/false positives, it would make sense that Anthropic would want to hide that by running it themselves as many times as they could until it actually generated some real, actionable results.

That can still be rather concerning when released to the public. If you have a d10,000 and only roll once, your chance of getting a nat 10k (high severity vulnerability) is really low. If you can roll it a million times, well now you've suddenly made the news.

I'm not in cybersecurity at all but I wouldn't be super surprised if at the end of the day AI ends up being much better optimized for finding vulnerabilities than writing and maintaining a large codebase for precisely this reason.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats