This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
AI safety & open models
Yesterday, Kimi K3's weights were published on HuggingFace. We now have an open model comparable to GPT 5.5 and Opus 4.8, which were SOTA only a couple months ago.
An example of what it can do on its own: this website that imitates macOS Desktop (background). Click around, every app has lots of features, and again, this was implemented in one shot. Do you not think that's impressive?
You can, at least in theory, run this on your own equipment. Millions of dollars of equipment, sure, but far more attainable than running GPT or Opus. A medium-sized corporation or small government can.
Which presents problems: its impressive capabilities can be used for evil, without guardrails or surveillance unlike GPT or Opus. Like Fable, Kimi has already found several vulnerabilites. For examples of real evil, see how other LLMs are being used for terrorism by Boko Haram (and almost certainly other groups).
Regulation concerns
Allegedly some US officials are considering restricting US companies from using Chinese open models. Despite this claim being repeated across many outlets, I didn't actually find any evidence. However, I did find plenty of tweets criticizing the models' development and themselves, like this tweet by Treasury Secretary Scott Bessent:
IP theft from Anthropic, who themselves are disregarding IP? Really?
What seems more indicative of a potential future restriction, the US government is already rushing unclear regulations for US models Fable/Mythos and GPT 5.6. If an open-weight model reaches somewhere around their capability, intuitively it would also be restricted, and Kimi is close.
Tech companies...support open weights models?
You have people like Dean Bell arguing for regulation, but many companies including Andressen Horowitz, Dell, IBM, Meta, Microsoft, and front and center Nvidia came out in support of open models, in this letter.
Key paragraphs (emphasis mine):
Also
Unlawful extraction from companies that have themselves unlawfully extracted? Again, really?
Regardless, I think overall it's a good sign.
Anthropic's position
Key quotes (emphasis theirs)
The important part is that Anthropic claims they don't want to ban open models, but want "mandatory safety testing" applied to all models. They use the threat of an AI-assisted superbug, which admittedly could cripple civilization, but so far is merely plausible. But that could effectively ban open models if it's implemented such that only closed models pass, like how "nobody can sleep under a bridge" applies equally to rich and poor but only affects the latter.
But it's also part of Plan A, proposed by the AI rationalists, who are supposed to be experts on this topic (what else have they been doing the past 10+ years?). Plan A actually argues against open models entirely, although it specifies that access to the models should be open, and all development and regulation discussions should be public. But how can we publicly develop models without making them open?
I'm curious what the Plan A authors think about Kimi K3, the open letter, and Anthropic's response; I haven't seen anything on lesswrong.com yet, although admittedly I only skimmed the front-page and recent.
My thoughts
For now, I support open weights models.
If someone comes up with a way to regulate AI development that doesn't eventually consolidate power into corrupt hands, sure. But who can be trusted? Even if future atrocities are caused by open models, they may be lighter than the atrocities committed in an alternative timeline, by a tyrant who gained power with the help of regulation, or lack of open models that prevented them.
As for "unlawful training": I still maintain the position that IP should gradually be completely abolished. AI training has already been ignoring IP, so I believe that should continue.
That includes China training on American models. I doubt China will surpass American companies if they're training on American models, especially since America has more hardware. And incentive? Come on, the American companies have enough incentive even if they had to distribute their models freely, from the dream of ASI.
One angle I rarely see discussed is that open weight models are inherently decelerationist for the frontier. They greatly reduce the profit motive to push the frontier, especially if distillation is allowed because the make it much harder to recoup the investment thus reducing the capital available to invest. Ultimately though they're definitionally unsafe and can never be made safe, any work put in to align them can be sanded off with post training and even with just mundane levels of uplift the idea that the offense/defense equilibrium will favor the defender in most areas, let alone all areas is just impossible to believe.
If you're just referring to different areas of software security, the long-term equilibrium is decidedly in favor of defense. Bugs, especially exploitable bugs, are a consequence of the fact that on any objective scale our programming languages still suck (especially the fact that the ones which suck less on security tend to suck more on performance) and our programmers suck (yeah, including me, sorry), not because it's actually impossible to write a program that does what you want but doesn't also let you get p0wned as soon as someone figures out just the right corrupt input data to send. In the meantime, before we have languages that aren't larded with Undefined Behavior pitfalls and/or the ability to cheaply write programs that never stumble into exploitable pitfalls, initiatives like Project Glasswing are a pretty good way to keep the short-term equilibrium in favor of the defender too.
If you're also thinking about e.g. biological security ... well, yeah, there is a chance we'll all be dead soon. I'd like to hope that, since our immune systems are pretty versatile, maybe natural pathogen evolution is already stress-testing us near the limit of what a bioweapon could do, and in the worst case maybe AI could quickly figure out a vaccine to prepare us in advance of infection by any even-more-dangerous inventions ... but "we can make software good enough" is practically a theorem, whereas "we can make immune systems good enough" is more of a prayer. That prayer would have to be answered, not just for humans directly, but for all the life that humans depend on. Florida's orange production is down over 90% since the first spread of "greening disease" there, and we've had decades of inability to fix it, and if something similarly unstoppable ever infects corn, rice, and/or wheat too then there are going to be a lot of starving people.
This is heavily disputed to put it lightly. The ask here is something like all software everywhere be written/rewritten incredibly defensively and continuously upgraded as the models get better at finding and exploiting vulnerabilities. Even hardening what we have now, even with mythos helping via glasswing, this is long term project. The trouble is that the defender has to win every fight in every domain for every surface, while the attacker needs to only win once. The effort disequilibrium is massive.
Yes, and I've personally read reports on my software that have gone through this program. Importantly the defends have better tools here.
Yes. Seems pretty bad. I personally do not want to die.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link