sarker
competency crisis actor
Suddenly I cannot remember the color of your eyes
Or the things we said as we stood together for the last time
User ID: 636
sarker
competency crisis actor
2hr ago
I thought they were the best?
While some are good people, they are on average not sending their best.
sarker
competency crisis actor
2hr ago
I would be very glad if it were "we need to make sure this doesn't help people develop cyber or biological weapons",
Oh, you are in luck! Amodei:
Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D that could accelerate these other risks.
AI companies that develop advanced AI models must have strong security standards that protect their model weights, should conduct regular red teaming and penetration testing, and should work with the government to defend against major threat actors.
Safety incidents in the four critical areas must be reported promptly.
Hopefully that clears things up.
sarker
competency crisis actor
2hr ago
And who advises on what the regulations should be?
Cor blimey, I never expected you'd come out as a libertarian! I didn't know you even had those in Ireland.
This is a fully general argument against regulation. If you think that AI in particular doesn't need regulation, let's discuss why.
It happened in this instance that Anthropic got slapped with precisely the kind of 'government regulating threats to public safety' that they had asked for in a mealy-mouthed way, and they didn't like the result when it was not in fact "no, our dear trusted special case AI, we will let you dictate terms to us when it comes to regulation, you are the special and shining exception to the rules".
What's the old saw? "My rules > your rules applied fairly > your rules applied unfairly"? This was not a fair application of the rules, and it has nothing to do with Anthropic needing to be an exception.
sarker
competency crisis actor
2hr ago
Then they both take the plane back home and keep advancing their models. Whatcha gonna do?
I am not interested in debating Plan A, but the implication that there is no proposal to audit compliance is false. This is a pretty low effort response.
AI 2040 is much closer to total disarmament than it is the NLPT, because the NLPT is about punching down (it is plainly true that the US/China could stop any non-nuclear power from advancing models) while AI 2040 / total disarmament is about restraining great powers from pursuing their interests, which is of course completely impossible unless you pull the nuclear card. The NLPT was completely useless at restraining the P5 from having as many nukes as they liked, after all, which is who we're actually worried will create models that are too powerful.
This just sounds like word games. Disarmament isn't when you stop building more nukes or improving existing nukes. Do I need to cite the dictionary?
You've set up an argument where actual nuclear disarmament must have been in the interests of the nuclear powers since it happened. Okay, well then if this happens then it's also in the interests of the great powers.
sarker
competency crisis actor
2hr ago
And I note that it happened with no regulation whatsoever.
Of course, that's only because Anthropic happened to be the first actor to reach this capability level and because they are, according to you, in a cult. It didn't have to turn out this way, and it may not turn out this way in the future.
This is jumping from the Motte you just presented to an incredibly speculative Bailey. Nobody has done this, there is no evidence that this is even possible now let alone when people have taken steps to prevent it.
I don't see why "this hasn't already happened" is a good reason to not take safety seriously.
I see no reason why the government with access to K7 should be outwitted by a lone maniac with access to K7. If it seems that way, by all means let's address the problem at hand rather than regulate people's ability to ask questions about knowledge the government doesn't want us to have.
It's not clear to me what you think biosecurity looks like in a world with highly intelligent AI that lacks safety guardrails, but it's difficult to imagine it isn't a panopticon of some sort.
If someone can demonstrate a lab process where a competently trained AI used competently nevertheless turns evil and attempts to cause damage even when made aware that this is not what its creators are asking for, I will take their arguments more seriously.
No "turning evil" is required here. All that's required is a human convincing the AI to help them with something that has evil ends. I hope you won't make me prove to you that humans can fool AIs?
And that's before we get into modifying the world's chips to no longer be able to run unapproved software and putting US/China killswitches in data centers.
You are, of course, conflating Anthropic's proposals with Plan A. They are not the same.
sarker
competency crisis actor
2hr ago
It is presently unclear if a junior engineer who has only ever used Claude can become a graybeard.
sarker
competency crisis actor
6hr ago
The fundamental difference is that the NLPT is about stopping weak states from getting nukes (which is possible because it is in the interest of great powers, who already got their nukes), while AI 2040 / total de-nuclearization requires that the great powers hold hands and sing kumbaya against each of their own interests.
The equivalent to AI 2040 but for nukes would be, because you're afraid of Putin deciding Après moi, le déluge and first-striking DC, to first strike Moscow pre-emptively to try and de-nuclearize him in advance. Obviously you can see the problems with this.
At this point there are only two countries with AI models near the frontier. If these countries make a deal that they will not advance their models past a certain capability level and will prevent other powers from doing so, this is much closer to the NLPT than disarmament. Most obviously, it does not entail any reduction in AI capabilities, where disarmament entails a reduction in nuclear capabilities.
sarker
competency crisis actor
6hr ago
I also work on AI. There's wide agreement that AI capabilities can be dangerous in the wrong hands unless the AI is aligned. We are now entering the time of AI actually being good at detecting security vulnerabilities in software. In the long run, firms will deploy their own AIs to probe their systems and keep ahead of attackers, but it would have been irresponsible for Anthropic to release a fully capable Mythos-powered zero day factory with no guardrails publicly without working with companies to address their security holes first.
In the medium term, I expect AIs will improve significantly in biological capabilities. It would be really quite bad if Kimi K7 helped some psycho develop a synthetic plague that killed a bunch of people, and it's worth taking steps to prevent such an outcome.
These are basic safety arguments that have been done to death that do not even require superintelligence, superpersuasion, or even a "will" on the part of the AI.
Anthropic's insistence on acting as a priesthood completely undermines their entirely-speculative case.
But it isn't solely Anthropic, you dismiss all safety concerns regardless of their source because you dismiss AI safety period.
sarker
competency crisis actor
6hr ago
That is quite clearly not what Corvos is arguing. If you have a concrete explanation of what in this regulatory framework doesn't take safety seriously, and what would, I'd be interested to hear it.
sarker
competency crisis actor
6hr ago
If somebody tells you out of the blue that you are going to die of cancer unless you take a 50g zinc tablet every day, it is obviously relevant if he is a travelling zinc salesman!
If somebody believed that there would be terrible consequences for the human race unless everyone took a 50g zinc tablet, could they be anything but a traveling zinc salesman, or a fellow, uh, traveler?
Your opinion is that we can't trust people concerned about AI safety about AI safety because it may increase their status. Okay, who can we trust about AI safety? People unconcerned about AI safety? Seems like epistemic closure to me.
That is, he has hidden his involvement to push his political manifesto whilst pretending to be an interested bystander
I can't honestly believe you think his involvement is "hidden" when he discloses it in the opening section of his first post on the topic.
Their intended solution is to get the two largest countries in the world to get together and fuck up anyone who disagrees with them in ways that would horrify them for literally any other problem except their super-special Millenarian one.
This is already basically how nuclear non-proliferation works, as we've seen in Iran. Again, you can argue about if AI is as dangerous as nuclear weapons, but if you assume arguendo it is, then this is an obviously similar regulatory regime that basically nobody opposes.
Your argument basically boils down to:
-
If AI really is dangerous, we would need to take drastic action to avoid bad outcomes.
-
Drastic action would be bad.
-
Therefore, AI can't be dangerous.
Or perhaps:
-
The people who think AI is dangerous are suggesting drastic action.
-
Therefore, AI can't be dangerous.
sarker
competency crisis actor
7hr ago
Did they intentionally bypass the cache, or did they not even know about it? It's not hard to believe that clueless programmers don't know that the value is in the cache.
The H1B code monkeys in Mumbai
Perhaps he's wondering why someone would sponsor a man for an H1B before employing him overseas.
sarker
competency crisis actor
7hr ago
It obviously can't be the case that "this increases the power of AI companies that take safety seriously" is an argument against a regulatory regime that takes AI safety seriously. The argument must actually be that AI safety is not a concern, not that even if it is a concern we must still be suicidally unconcerned.
But, on the other hand, did it really change everything? Perhaps not. People still work, get sick, and die, same as they always did. I guess if that changes then we’ll really be in new territory.
Amazing that the invention of fire or even the advent of intelligent life on this planet (using "people" loosely) didn't 'change everything' by this definition. Tough crowd!
sarker
competency crisis actor
7hr ago
where careless devs don't do even the most basic things to make the computer not have to do the same work twice.
Exactly, because it's basically not worth it outside of actually high performance computing. Who's not going to use a product because it's 20% slower but otherwise better than competitors? Vim is way faster than Microsoft Word, but who cares?
It's especially embarrassing when the shittier way of doing it is harder to write than the non-pessimized way and retarded web dev monkeys do it anyways.
You'll have to provide some examples.
sarker
competency crisis actor
7hr ago
Anthropic's conception of 'mitigating AI risks' is 'do everything I say, just the way I say it, and this system has to be enforced by someone who thinks like me'.
This is a fully general counterargument against any regulation. Even your proposed regime rounds off to "do everything I say the way I say it and it has to be enforced by someone who thinks like me."
If your preferred regulatory regime also excludes almost all your potential competitors except the ones you're basically okay with, then it is indistinguishable from a power grab.
Their stated beliefs on AI dangers, which philosophically long predate Anthropic as a going concern, are entirely consistent with the proposed regime. In fact, it would be contradictory for them to believe in AI risks and argue for laissez-faire. Perhaps you disagree on the risks, but that is the root of the disagreement, not the regulatory framework.
If companies are still employing software engineers (and they all are) then I’d say we’re mostly in “nothing happens” territory.
It is too early to be confident. Many people in FAANG have no idea how to use AI effectively and many others refuse on principle. Massive layoffs (20%+) are unlikely due to reasons of political economy despite the staggering amount of deadwood at these companies. It will take a long time for these ships to turn, we are still in the early stages of diffusion.
But if people are still going to work then, how different are things, really?
People went to work before the industrial revolution and people went to work after. Perhaps your opinion is that "nothing happened" during the industrial revolution; let's say I disagree.
sarker
competency crisis actor
7hr ago
You can, but cars, airplanes and drugs are famous examples of very heavily regulated industries where creating a new business is almost impossible, incumbents rule, and incumbents and their lobbyists design the majority of the regulations themselves.
Creating a new business in these sectors is difficult because the sectors are inherently difficult to make money in. Moderna was founded in 2010 and basically had nothing going on until Covid. SpaceX was founded 2002 and only had a successful launch in 2008 after near bankruptcy. It is nevertheless possible to enter these markets.
If the Trump government had calmly and consistently mandated open weight releases and anti-monopoly legislation, Anthropic would be screaming about how this was unsafe.
I don't see the contradiction here. "You say you want regulatory regime A, but in fact you would hate regulatory regime B which doesn't accomplish the same goals. Curious!" The point is not that Anthropic wants a regulatory regime, they want a regulatory regime that is oriented around mitigating AI risks.
sarker
competency crisis actor
8hr ago
"We will create super-intelligence, and then the thing magically becomes alive just like us, so just like us it will have goals and aims of its own, and we have to make sure it is well-instructed in How To Be Nice Liberal and then it can take care of us like pampered pedigree cats as it colonises the known universe".
How about "we need to make sure the thing doesn't help people develop cyber weapons or biological weapons?"
sarker
competency crisis actor
8hr ago
They want to be the ones deciding who gets to play with the toys and the government tamely follows their instructions as regards shutting down their rivals, but it's not going to work like that.
I don't know why people keep repeating this. From Anthropic:
It is time to go beyond transparency to more serious and binding regulation of AI. I believe the best analogy, at least at the current stage of the exponential, is to cars, airplanes, or drugs—powerful technologies essential to the modern economy, but capable of killing large numbers of people if designed or operated poorly. I therefore believe we should model AI regulation on agencies like the Federal Aviation Administration (FAA). Frontier AI models, like airplanes, should be required to go through technical testing and auditing, and their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety. I am grateful to see the Trump administration’s Executive Order move incrementally towards a greater role for government in AI, though Anthropic’s proposal recommends even further action.
Does this sound like Anthropic decides who uses their models? Does Ford decide who uses their cars?
Yes, they weren't happy about a schizophrenic and vindictive ban of their model due to capabilities that it shares with other models.
You can be in favor of a consistent regulatory framework without being in favor of arbitrary decisions by the president based on who is in and who is out.
sarker
competency crisis actor
8hr ago
For example, you currently can't simply ask "Hello Mr. Claude the site latency is too high, please fix it,"
Given access to logs, metrics, etc. I expect Claude would in fact be able to fix this.
Capability hits a wall: For the greybeards, nothing happens, for the kids, those who choose the manual route will come out ahead.
We have already exited the "nothing happens" condition given present capabilities. The graybeards are much better off acting in a staff+ capacity using LLMs than writing code themselves. The question of what to do with junior engineers is an interesting one.
sarker
competency crisis actor
8hr ago
That's largely a casualty of Moore's law. There's no point in optimizing candy crush to run on the Apollo guidance computer when nobody is going to run it on hardware like that.
sarker
competency crisis actor
20hr ago
I also agree, though I don't know about the rambliness. SMH could never be accused of being concise. I do feel that the quality of the comments has been worse.
On a lark, I got hy3 (free on openrouter!) to scrape @self_made_human's comments and graph length and score. I don't see a clear trend in length, but it does seem like there's been more comments with zero or negative scores since late 2025/early 2026.
It does make it presently unclear in a general sense.
Then nothing. Do you want a cookie?
More options
Context Copy link