aqouta
Friends:
User ID: 75
It seems pretty plausible that the frontier labs could fail simply because demand doesn't meet their projections...
This is basically describing what I mean by if scaling stops tomorrow. It can certainly be argued but it's my contention that if things keep scaling like this and all else holds steady the labs could just start directly eating segments of the economy. The "application layer" guys seem to expect the labs to just lay down and let them pay commodity prices to stand up a thin wrapper. But at a certain point anthropic can just instruct opus 7 "go ahead and stand up any product that would be profitable to run by a wrapper company, make no mistakes". That's a bit of a tongue in cheek example but what is stopping anthropic with a sufficiently advanced unreleased model from just becoming a hedge fund and doing to the financial industry what they're doing to mathematics? They have a wetlab, there's a lot of money in drug discovery.
I'm not making a strong claim here, I don't know what even the near future will bring. I don't think the labs failing is even all that unlikely. But I also don't think they'll have trouble finding investors money if they ask for it and it's not obvious that scaling laws have failed. I've personally set aside like a quarter of my net worth to be ready to invest in ant and oai at IPO.
But we're in a subsidized era of AI right now, and it may look very different once that subsidy ends.
I'll poke at the term subsidy just like the term bubble. Research is being subsidized, the inference is not. And it's hard to tell the degree of the research subsidy.
If they actually revise their numbers back up to $1.4 trillion in 2030 and meet that infrastructure goal, feel free to ping me and I will agree that this was a bad example. Then I will point you to the 2027 Project where it postulates that the robots would kill us all by now, as my fallback example.
If you've been keeping score the 2027 project is doing much better than its detractors and pretty damn good in absolute terms.
aqouta
20hr ago
The word "bubble" is hurting more than it's helping here. It's beyond question that the money spent on training frontier models will produce an absolutely massive amount of value for the world. But it's still to be seen how much of that value can be captured by the labs themselves. This isn't really the same shaped as tulip mania or people paying absurd amounts of money for pets.com. If the labs fail economically(Which I think could only happen if scaling laws collapse roughly tomorrow) then the weights will still be around and still worth quite a bit to serve on the hardware which it will still be around and be worth it to keep running inference on. The labs could fail but they wouldn't be failing because AI is overhyped or a scam, just that running a research company in a competitive environment without state granted monopolies on the produce of your research is a brutal business to be in.
aqouta
22hr ago
You just have to believe that AI is an existential risk and now that's more important than charity.
I don't know why you and the @quiet_NaN are accusing Scott of being uncharitable. His open letter was not uncharitable, it was certainly aggressive and impolite but charitability would imply Scott was making unfair assumptions about Pinker's arguments which he didn't do as far as I can tell. Scott has been vicious in the past plenty of times.
aqouta
3d ago
As I said, "This does not necessarily mean that regulations are bad." I've been trying to figure out your position, which seems to be that regulation wouldn't hurt the little guys but would cost them money but it's not a regulatory moat. It seems quite possible that regulations can be good and be a regulatory moat, there's no contradiction there. But you've been arguing that they wouldn't serve as a moat, not that they are good or bad.
Yes, I don't think they'd be a moat. It's up in the air, as they're not strongly defined, if they'd net help or hinder catch up, as written I would guess they would help catch up. Slowing the front runners down gives those behind to fast follow at leisure. This is actually a negative in a sense to me because it lets chinese labs catch up relatively and there are world where that is quite bad by my values.
Okay, see this makes sense to me! Thank you! (Sincerely!) I don't think you would have gotten so many downvotes if you had said "a16z thinks regulation of the frontier is a camel's nose under the tent to ban their shady services that have been implicated in suicide and cheating.* I appreciate you taking the time to lay that out for me.
I don't like to make the arguments from incentives on this subject because I don't think they're the right frame of the argument. It's the frame the a16z/accelerationists want to push because then it's a matter of their interests vs anthropic and don't you really care about freedom of speech ect ect? But I don't think the important point here is about who's interests should prevail, even if that's also for contingent circumstantial reasons an argument my side can win against them. Because even if the balanced tipped and from interests alone the labs came out looking worse through this cynical lens I still don't want everyone to die. If entrenching AI control into a handful of companies actually was the only way to keep everyone from dying then we should do that, albeit probably it would be best to nationalize them in that case. I don't think that's the case but I don't want to hinge my defense of regulations on this unstable ground.
aqouta
3d ago
The automated CW threads are a matter of our history, we trace our lineage back to Scott's weekly culture war threads when he had them on SSC before the CW got hot enough that they were pushed first to his subreddit and then off the subreddit to /r/themotte, finally we left reddit with some help from the rdrama.net fork of ruqqus. Rdrama.net probably belongs in your cataloguing of far flung communities. The weekly threads are intentionally where the culture war mainly is and that's the thing people come here for. Sometimes we post to the main wall but for instance your post would probably get more traction as a top level CW thread in the main weekly.
aqouta
3d ago
Your position upstream was "it's simply not the case that they give the labs asking for them a competitive advantage" and then you asked me to "point to a regulation proposed by any of the labs that would actually hurt the little guys more than the frontier labs? Or at all?" Seems like you can create a competitive advantage without locking down the entire space for yourself.
And you didn't, you refused to do any work so for the conversation to continue I cited the proposed legislation myself which does not hurt the little guys more than the big guys.
As we discussed previously, you can hit the FLOPs requirement to trigger oversight without spending billions; Mistral did it. Now, it is true that Anthropic's proposed rules would also not apply to anyone who made less than $500 million in annual AI-derived revenue or spend more than $1 billion on R&D annually, which doesn't seem to have anything to do with the frontier at all - if someone decided to make an Evil AI in their basement/supervillain lair based on an existing open-sourced design, it wouldn't need to comply with Anthropic's model regulations, even if it exceeded the FLOPs requirement (which some open-source models do).
the openai FLOPs requirement is 10^26 which cuts a bit higher, but if your argument was "actually it should be 10^27" then we can have a conversation. But you don't seem at all interested in engaging with regulation as if it's a valid end at all. You make arguments that generalize to all regulation in all cases is bad because you can make up a story for how this somehow helps incumbent companies.
If I am getting your argument right, then, application layer startups are against regulations less because it might affect them directly (in terms of oversight) and more that it will slow down the progress of the frontier models that they interface with to make their "absolute rancid slop," thus slowing their revenue gain. Is that a good summary?
They're part of the a16z group that broadly opposes all regulation and actively funds pacs to this effect(leading the future). Their opposition to this regulation in particular is over determined. They want the frontier to advance quickly and be distilled by open weight models because it's their belief that every other layer will be a commodity layer and they're the real enduring layer. They have a love hate relationship with the frontier labs. They also oppose all regulation in general because they fear if some AI regulatory organization got some teeth they might come after their slop, which the public widely hates. for fun here are a few examples of a16z funded projects:
Doublespeed Slop social media spammatron 5000
We've built an attention intelligence platform to build and deploy social agents at scale across every demographic and niche. Capturing mindshare on organic social is currently a human labor problem, instead our autonomous creators learn from every interaction, and become more valuable over time driving conversions and engaging with millions 24/7.
Cluely literally cheating as a service
Cluely, a startup that claims to help users “cheat” on job interviews, exams, and sales calls, has raised a $15 million Series A led by Andreessen Horowitz, the company announced on Friday with a video posted on X.
Character.AI this one literally has blood on its hands
In November 2023, 13-year-old Juliana Peralta of Colorado died by suicide after extensive interactions with multiple chatbots on Character.ai. She primarily confided suicidal thoughts and mental health struggles in a chatbot based on the character Hero from the video game Omori, while also engaging in sexually explicit conversations—often initiated by the bots—with others, including those based on characters from children's series such as Harry Potter.
aqouta
3d ago
Yes, none should probably advance much further, at least in this form, until they can collectively get that under control. If the regulations freeze the frontier then that seems good for the little guys trying to catch up.
aqouta
3d ago
Then they'd have been exempted from the regulation or have to do a fairly small number of things like submit the model to review before deploying it. It just really don't seem like a huge barrier when you already have to be spending a billion dollars to even qualify for regulation.
aqouta
3d ago
If you have to run a race where the leader is 5 km ahead with training weights on, and you also have to put on training weights once you get within 1 km, it sounds like something that will severely slow down your ability to catch up. Maybe not for ever, but for long enough to count for the mortal leadership.
I fail to see how this meaningfully is different to trying to run that same race where no one wears safety equipment. And comparing basic safety and verification checks to a 5kg vest is silly. They kind of company that can meaningfully challenge the frontier is like oracle sized or a new startup with industry insiders and several billion dollars in funding. There just really can't be a small guy structurally when you have to bring billions of dollars to the table anyways.
And while "we could all die" would be very unfortunate, the probability is not yet above my personal willing-to-be-Pascal-mugged threshold. Especially given that we will all die in due time unless sufficiently fast progress in medicine makes it otherwise, and frontier labs being mired in regulation sounds like the opposite of fast progress.
Just the diffusion of what we already have will take a lot of time and I think advance medicine quite quickly. And no one is proposing preventing narrow drug research models. This isn't an either or thing. This is trying to avoid going supercritical. We really don't need to roll these dice to get radical advances.
aqouta
3d ago
Can you even explain how the tangible results give them an advantage? The best story I can come up with is that they're able to comply with the regulations better somehow because they were already big. But it's hard for me to imagine a startup lab that is able to get billions of dollars together for a training run that can't figure out how to follow regulation. And the actual tangible direct effect of the regulation is that they stop pushing so far ahead, which just obviously gives other labs a chance to catch up as a first order effect.
Even if I grant that it gave them some unforeseen or specifically articulated advantage surely "we could all die" is enough reason to at least make an attempt that we can modify later if someone does find some harm later? It's not like anyone is saying these rules should be written in stone and assumed perfect for all time, we're trying to coordinate a slow down and ensure safe practices. This is a red hot industry at the center of discourse, it's not likely that any harms will go unnoticed or discussed.
aqouta
3d ago
Usually, when a scenario can be explained by "huge corporations conspire to corner the market" I would rather look for evidence that they couldn't do that rather than that they did it. Because my first question is "why wouldn't they?".
Ok so look for it. Did you find any? Do you think it's categorically impossible that driving forward into Recursive self improvement in a race condition could be dangerous? It's very exhausting, you have a motive and are in search for a crime, but not a real search because you don't bother to actually look. Meanwhile we might be driving off a cliff.
aqouta
3d ago
As per our discussion last month, it seems pretty clear that Anthropic's proposed regulations would hit "little guys" like Mistral, and their regulations impose a burden on small companies, burdens are costly, cost hurts, ergo is seems pretty clear that the regulations would hurt little guys. (This does not necessarily mean that regulations are bad.)
Ok, so Mistral would be required to to submit to the security requirements that the other labs have to submit to. Such as registering with CAISI or some other organization depending on what blueprint you're talking about and submit their models to pre-release testing. How does this lock them out of the frontier exactly? I actually wouldn't be surprised if they already had stricter requirements from the EU given they're based in france.
With that being said, I'm still not clear on your argument. If your position is that the regulations wouldn't hurt the little guys, or would hurt the frontier labs (whom you say they hate) more so than the little guys, improving the competitive position of the little guys, then why are the little guys against the regulations?
They want the labs to push the frontier forward as fast as possible, the labs want to slow down. They also find it annoying when the labs have guardrails at all. There's two things "the little guys" could mean, smaller labs and "application layer" startups. The concern for the smaller labs seems totally hallucinated, the idea that a lab is going to be able to put billions into training runs but can't comply with some oversight is absurd on its face. The real concern is about these "application layer" startups who are shoveling absolute rancid slop everywhere as fast as they can. These slop merchants don't discriminate on what type of regulations are proposed for AI, they oppose it all categorically.
aqouta
3d ago
They didn't originate from bills, but from my understanding of AI safety discourse, the version of that discourse being presented by the lab leadership, their calls for regulation, their market position, and the general way they've been generating "scary" AI incidents that don't seem organic and do seem to bolster their calls for "regulation".
So a narrative story you've been told that you don't feel a need to ground out in any actual actions taken by any of the people or groups being discussed? They're not intentionally committing felonies, that doesn't make sense. The downsides on that are wildly out of proportion to any imagined upsides.
...how is this not a moat?
How is it a moat? If you're spending less than a billion dollars then you're entirely exempt and if you're spending more than that then you just have to comply with simple things like pre-release testing by a third party that all the frontier labs have to comply with. How could you possibly design a regulatory system that is less moat shaped than that?
aqouta
3d ago
Do you have any actual evidence of this? The cartel accusation as far as I can source it comes from them asking for an exemption to the Sherman act so that they can work together on standards, which is common among many industries and would not at all form any kind of moat. There's a lot of accusations being thrown around which as far as I can tell are entirely hallucinated.
what is an a16z type person?
Andreessen Horowitz circle, I outline the perspective here.
More generally, I have a fairly embedded prior that the AI companies are in fact actively attempting to secure a regulatory moat, wherein they are the only companies legally permitted to provide AI services. Actions they've taken, particularly the "rogue" hacks, appear to me to fit this model quite well.
Am I being lied to? which bills are supposed to disprove this prior, and why?
From which bills did the priors originate from? Openai and Anthropic have both published blueprint suggestions that explicitly exempt anyone doing training runs below 10^25 FLOP earn less than $500M in AI revenue or spend less than $1B a year on AI R&D. If the worry is that the regulations would create a moat we could actually talk about the regulations and demand they be written such that a moat wouldn't be created. But the people pushing this point have not read any of the proposals and do not care about the moat argument besides its use as a soldier.
They don't want any regulation ever or for things to slow down, more broadly they hate the frontier labs in general. There are a few groups in this category, some are angry that they passed on investing in the labs early on in favor of crypto. Broadly their preferred future looks something like the labs burning the money of other investors to produce powerful models that chinese labs are allowed to distill and then they can run them locally to power their anti-social revenue extracting products. They see themselves as the "application layer" that Huang often talks about. They have very little vision about the future of AI and broadly only care about what makes their low effort wrapper start up more money.
Can you explain the alternative view? Can you point to a regulation proposed by any of the labs that would actually hurt the little guys more than the frontier labs? Or at all?
The vote difference with no actual rebuttal is pretty funny. People you trust are lying to you. You should not trust them. These are lies cooked up by A16z type people because they want to be able to produce their slop gambling porn simulators more cheaply. That's the faction who's lies you're swallowing and you're getting mad at people for pointing it out.
You can look up the proposed bills, it's simply not the case that they give the labs asking for them a competitive advantage. It is not an opinion. It is untrue. It is a lie.
aqouta
4d ago
He also co-wrote big yud's "if anyone builds it everyone dies" book. He's not a big blogging voice but very much in the circle.
aqouta
4d ago
as one of the broad reasons for calling for regulation was to get a legally enforced competitive advantage
This being entirely untrue must have been explained to you before, why spread the lie?
aqouta
4d ago
I'm sorry but "they're fighting these people with insidious methods but they're bad at it" is such a retarded line of attack. These people are running some of the most valuable companies in the world. You have to do better than this.
aqouta
5d ago
There isn't an argument being made. "a contemptable language game" come on man, he was pointing out some amusing parallels. That people who hate him and his movement are blowing it out of all proportion
Christians' moral objections to Aella's activities and to the Yudkowskiites' exploitation of them for tactical advantage
Tactical advantage? What are you talking about dude? You already said this is not to his advantage. This isn't some weird 10 diminsional chess game, they're just abnormal people. They're the type of people who take thought experiments seriously and follow them to their conclusions. Turns out doing that a lot lets you make some pretty surprisingly accurate predictions about how important things like how AI would go. They also take things like liberal sexual ethics to their logical conclusion, which you oppose but isn't really an argument against their other predictions.
I do not accept that the Yudkowskiites or the "Effective Altruism" movement they have parasitized
parasitized? They birthed the movement.
aqouta
6d ago
The context is that there is kind of a civil war in Scott's general sphere of twitter over the AI saftey question. Scott, having a history of being tolerant of religious rightists during the culture wars, has a wide number of Christians within his social web. These Christians and rightists variable side with or against his positions but often remain civil. It wasn't obvious that they were going to take the anti-ai-safety position but it's starting to seem like the worm is turning that way and there are a lot of attacks on EA/Rationalists/Safetiest from the Religious right direction. The subtext is kind of clear "We know we're weird by today's standards and but were you and your people not once weird by today's standards?" He's being accused of doing the meme but scott has engaged earnestly and in good faith the christians frequently enough that I don't think you can accusing him of cynicism here. He does in fact think it's admirable that Jesus forgave prostitutes and preached against a corrupt moral order, in many ways his movement is downstream of christian enlightenment.
aqouta
6d ago
My response to both of your points is the same. Unless you specifically prompt them otherwise the output from frontier models is always very hedged, which is why it half agreed with MKC without being willing to really fall on one side. You're taking a hedge as disagreement, in reality it neither endorses nor disagrees with MK, which is a fine null hypothesis. Where it doesn't hedge and outright states something is false or true it's pretty trustworthy unless you have strong evidence.
Zvi calls this concept bounded mistrust and it applies to all media sources. They can mislead but almost never lie.
As I said, if the scaling stops tomorrow then maybe the models simply are not good enough to do this autonomously and other human labor is required to guide them along. If it continues at this rate for another year or two things change dramatically.
I don't have time right now to rehash this debate again. using sonnet instead of opus, not having a proper harness and planning/review cycle. I've seen the thing go, I've put out projects in weeks that would have taken months. If you don't believe it has the juice then so be it.
You're getting pennies of sonnet for free, I'm burning through $3k+ in tokens a month. Although the psychology here is confusing to me. You'd have to consider your time basically worthless to avoid playing $20/month for some opus 5.5 usage instead of sonnet 5.
More options
Context Copy link