BANNED USER: temperamental dysregulation
DaseindustriesLtd
late version of a small language model
Tell me about it.
User ID: 745
Banned by: @Amadan
DaseindustriesLtd
late version of a small language model
1yr ago
Putin would have caved because his nation is barely hanging on while fighting against a 3rd rate local power
As a matter of fact I think this is not how we should be perceiving Ukraine, and in the present condition it would likely have been able to overwhelm any European military except perhaps France and Poland one on one. Consider that Europeans are not actually Aryan superhumans, their pretty exercises would amount to meme material in a week of fighting a real large scale war, and they have very little in the way of materiel too. They are concerned about Russia for a good reason: they are in fact weak.
DaseindustriesLtd
late version of a small language model
1yr ago
That's fine, I don't feel entitled to your time at all. I also can't predict what might trigger you, just like you cannot predict what would trigger me, nor does it seems like you would care.
Fetishizing algorithmic design is, I think, a sign of mediocre understanding of LLMs, being enthralled by cleverness. Data engineering carves more interesting structure into weighs.
The discussion was originally about labs overwhelmingly focused on LLMs and competing for top talent in all of ML industry so partially that was just me speaking loosely.
I do in fact agree with heads of those labs and most star researchers they've got that LLMs strikingly similar to what was found in 201 7 will suffice for the shortest, even if not the globally optimal route to “AGI” (it's an economic concept now anyway, apparently). But it is fair that in terms of basic research there are bigger, greener pastures of intellectual inquiry, and who knows - maybe we will even find something more general and scalable than a well-designed Transformer there. Then again, my view is that taste is to be calibrated to the best current estimate of the promise of available directions, and in conjunction with the above this leads me to a strong opinion on people who dismiss work around Transformers, chiefly work on training signal sources that I've covered above, as “not new science”. Fuck it, it is science, even if a bit of a different discipline. You don't own the concept, what is this infuriatingly infantile dick-measuring?
It's not so much that I hold non-LLM, non-Transformer-centric algo design work in contempt as I am irritated by their own smug, egocentric condescension towards what I see as the royal road. Contrarianism, especially defensive contrarianism, is often obnoxious.
DaseindustriesLtd
late version of a small language model
1yr ago
We don't know how Gemini is made. At this point I assume it's something incredibly dumb like Noam Shazeer's reduced attention schemes and not, say, DeepSeek's NSA. In short though, attention inherently allows for sparsity.
DaseindustriesLtd
late version of a small language model
1yr ago
Once again I notice that I am usually right in being rude to people, as their responses demonstrate precisely the character flaws inferred. This is a low-content post in defense of wounded ego, with snappy Marvel-esque one-liners («Won a Nobel prize») and dunks optimized for the audience but next to no interest in engagement on the object level. Yes, ML != LLMs, so what? Are you not talking about Altman and Elon who both clearly put their chips on LLMs? «That was a joke», yeah I get your jokes. Here's one you missed:
data engineering is needed and important! But it's not revolutionary. Data engineering, is the same as its always been.
It's not the same, though, that's the thing. Returning back to my point that has upset you –
Fetishizing algorithmic design is, I think, a sign of mediocre understanding of ML, being enthralled by cleverness. Data engineering carves more interesting structure into weighs.
– I meant concretely that this is why leading companies now prioritize creation of training signal sources, that is: datasets themselves (filtered web corpora, enriched and paraphrased data, purely synthetic data, even entirely non-lingual data with properties that induce interesting behaviors), curricula of datasets, model merging and distillation methods, training environments and reward shaping – over basic architecture research, in terms of non-compute spend and researcher hours; under the (rational, I believe) assumption that this has higher ROI for the ultimate goal of reaching "AGI", and that its fruit will be readily applicable to whatever future algorithmic progress may yield. This goes far beyond ScaleAI's efforts in harnessing Pinoy Intelligence to annotate samples for RLHF and you have not even bothered to address any of this. If you think names of Old Titans are a valid argument, I present Hutter as someone who Gets It, gets that what you have a sufficiently general architecture to approximate is at our stage more interesting in terms of eventual structure than how you achieve this potential generality.
This older paper is a neat illustration too. Sohl-Dickstein and Metz have done a little bit of work in non-LLM algo design if you recall, maybe you'll recognize at least them as half-decent scientists.
Now, as regards poor taste in intellectual disagreements, let's revisit this:
Regardless of whether transformers are a dead-end or not, the current approach isn't doing new science or algo design. Its throwing more and more compute at the problem and then doing the Deepseek approach of finetuning the assembly level gpu instructions to exploit the compute even better so you can throw more compute at it. I doubt, Hinton, Goodfellow, LeCunn, Schimdhubber et al. have any desire to do that. Maybe if xAI did something revolutionary like leave the LLM space or introduce a non-MoE-Transformer model for AGI, then talent of that caliber might want to work there. Currently they exist so Elon can piss all over Altman.
My rudeness was not unprovoked; it was informed by the bolded parts. I saw it as a hubristic, elitist, oblivious, tone-deaf insult towards people – scientists – actually moving the field forward today, rather than 8 or 28 years ago, and I do not care that it's slightly obfuscated or that you lack self-awareness to recognize the douche in the mirror but are eager to chimp out at it as you currently do.
Likewise my entire point, before you jumped into insult me, is that the Big Names in ML/AI are "fetishy algo freaks" They shockingly don't want to do non "mediocre algo butt sniffing" work. And Data Engineering isn't new, it isn't revolutionary, it's great, it works well, but it doesn't require some 1% ML researcher to pull it off. It requires a solid engineering team, some technical know-how, and a willingness to get your hands dirty. But no one is going to get famous doing it. It's an engineering task not a research task. And since research tasks are what people pay the ludicrously big bucks for at tech companies the engineers at xAI aren't being paid some massive king-sized salary...
yes thanks for clarification, that's exactly as I understood you.
I claim that to the extent that «talent of that caliber» shares your conceit that design of clever new algorithmic primitives for ANNs is «exciting new science» whereas data work remains and will remain ScaleAI tier «mere data engineering, same as always», this talent is behind the times, too set in their ways, and is resting on its laurels; indeed this is the same high-level philosophical error or prizing manual structure design over simplicity, generality and scalability that keeps repeating on every revolution in AI, and that Sutton has famously exposed. They are free to work on whatever excites them, publish cute papers for fellow affocionados where they beat untuned mainstream baselines, or just leave the frontlines altogether, and even loudly assert that they have superior taste if they so choose, which in my view is just irrational fetishism plus inflamed ego; I think taste is to be calibrated to actual promise of directions. But indeed, what do I know. You are free to share their presumptions. New scientific talent will figure it out.
Seems like you are asking us to praise ignorance over discovery?
To me it seems like the opposite, we just disagree on what qualifies as discovery or science at all, due to differences in taste.
As an exercise, can you tell me THE engineer at Deepseek who proposed or wrote their Parallel Thread Execution(PTX) code with a citation?
Egoists gonna be egoists.
Zhean Xu probably. But I think everyone on (Chenggang Zhao and Shangyan Zhou and Liyue Zhang and Chengqi Deng and Zhean Xu and Yuxuan Liu and Kuai Yu and Jiashi Li and Liang Zhao) list could ask for a megabuck total comp in a frontier lab now, and expect affirmative response.
I see you took this pretty personally.
All I have to say is that top AI research companies (not ScaleAI) are already doing data engineering (expansively understood to include training signal source) and this is the most well-guarded part of the stack, everything else they share more willingly. Data curation, curricula, and yes, human annotation are a giant chunk of what they do. I've seen Anthropic RLHF data, it's very labor intensive and it instantly becomes clear why Sonnet is so much better than its competitors.
They clearly enjoy designing "algos", and the world clearly respects them greatly for that expertise.
Really glad for them and the world.
Past glory is no evidence of current correctness, however. LeCun with his «AR-LLMs suck» has made himself a lolcow, so has Schimidhuber. Hochreiter has spent the last few years trying to one-up the Transformer and fell to the usual «untuned baseline» issue, miserably. Meta keeps churning out papers on architectures; they got spooked by DeepSeek V3 which architecture section opens with «The basic architecture of DeepSeek-V3 is still within the Transformer (Vaswani et al., 2017) framework» and decided to rework the whole Llama 4 stack. Tri Dao did incredibly hard work with Mamba 1/2 and where is Mamba? In models that fall apart on any long context eval more rigorous than NIAH. Google published Griffin/Hawk because it's not valuable enough to hide. What has Hinton done recently, Forward-Forward? Friston tried his hand at this with EBMs and seems to have degraded into pure grift. Shazeer's last works are just «transformers but less attention» and it works fine. What's Goodfellow up to? More fundamental architecture search is becoming the domain of mentally ill 17yo twitter anons.
The most significant real advances in it are driven by what you also condescendingly dismiss – «low-level Cuda compiler writing and server orchestration», or rather hardware-aware Transformer redesigns for greater scalability and unit economics, see DeepSeek's NSA paper.
This Transformer is just a paltry, fetish, "algo".
Transformer training is easy to parallelize and it's expressive enough. Incentives to find anything substantially better increase by OOM year on year, so does the compute and labor spent on it, to no discernible result. I think it's time to let go of faulty analogies and accept the most likely reality.
DaseindustriesLtd
late version of a small language model
1yr ago
Regardless of whether transformers are a dead-end or not, the current approach isn't doing new science or algo design. Its throwing more and more compute at the problem
Fetishizing algorithmic design is, I think, a sign of mediocre understanding of ML, being enthralled by cleverness. Data engineering carves more interesting structure into weighs.
DaseindustriesLtd
late version of a small language model
1yr ago
Mistral was an upstart lab formed by Meta and GDM alumni. We can look up bios of DeepSeek, they are youngsters with no experience in pretraining large language models and no connection to frontier labs, most have never worked nor studied outside China (according to CEO, V2 model is all indigenous). They strongly emphasize their disinterest in your prior experience in hiring. You can consider this all clever spectacle but I think it's time to admit they're just that good.
By the way, they're starting to opensource their infrastructure on Monday.
DaseindustriesLtd
late version of a small language model
1yr ago
Conspiracy theories around DeepSeek are pretty funny, people twist themselves into pretzels to not acknowledge the most parsimonious hypothesis. Because it feels too wild, I guess. Maybe too scary as well, because it suggests that China can birth like a hundred more such companies if it finds a hundred such CEOs. I collect these stories. They've used all of Singapore's compute! They pay $1.3M to top researchers! It's a «choreographed emergence» to deceive the oh-so-important Dean (but he knows better than to trust ChiComs)! The scale is much bigger, there are hidden disciples in cloistered cultivation! It's all so very creative.
In any case I have the direct opposite impression about papers. They are kids, overwhelmingly under 30 and often 20, and they write very naturally and not academically. It's just raw intelligence and curiosity, not experience. It is known that roughly everyone at DeepSeek speaks fluent English – not normal for Chinese labs; they pay extreme attention to culture fit and aptitude in recruiting, and are severely ageist. Many core innovations come from undergrad interns; the first author on the NSA paper is an intern with anime pfp too. We have reports from competitors' employees who had rejected their offer because they perceived the company as too small and weak for the declared ambition. I don't know how to break it to you, but there's no Iron Curtain, things are fairly transparent.
These are the best ML PhDs China has.
Maybe. I'm not sure if DeepSeek has even 50 Ph.Ds though, and ByteDance has thousands.
If your hypothesis is correct, they will not significantly accelerate, now that they're acknowledged as national champions and are offered OOMs more resources than they had before. I think they will.
DaseindustriesLtd
late version of a small language model
1yr ago
Grok 3 is quite certainly the best among available Instruct models. Grok-Mini is on par with o3-mini, PR nonsense about low-medium-high differentiation aside (DeepSeek docs suggest it's just something like an inference parameter for length of reasoning, so we should not assume that o3-mini-high is a superior quality model). Grok's RL training has only just begun, it is probable that they've reassembled their pipeline after DeepSeek-R1 paper came out.
Every lab will accelerate significantly in the near future so I wouldn't overindex on this leap. Elon just wanted to claim priority at least for a while, much like DeepSeek's Wenfeng, so both rushed to ship a next-generation project faster than competition, but fundamentals are still in favor of Google, Meta (despite horrible self-defeating culture) and OpenAI given Starship program. Grok 3 is just the first in a large crop of models trained on 100K+ class GPU fleets. Orion, Llama 4, Claude-Next are coming.
Even so, we have definitely received another frontier actor.
DaseindustriesLtd
late version of a small language model
1yr ago
With a few weeks of space between the initial marketing hype and observation, and Deepseek seems to be most notable for (a) claiming to have taken less money to develop (which is unclear given the nature of China subsidies), (b) being built off of other tech (which helps explain (a), and (c) being relatively cheap (which is partially explained by (a).
Man, you're really committed to the bit of an old spook who disdains inspecting the object level because you can't trust nuthin' out of Red Choyna. I could just go watch this guy or his pal instead.
It wouln't be an exaggeration to say that the technical ML community has acknowledged DeepSeek as the most impressive lab currently active, pound for pound. What subsidies, there is nothing to subsidize, except perhaps sharashkas with hidden geniuses.
DaseindustriesLtd
late version of a small language model
1yr ago
The US of A was founded by Emma Lazarus in 1965, or so I'm told.
DaseindustriesLtd
late version of a small language model
1yr ago
This is definitely true, but are you really expecting good results from collapsing the whole house of polite fictions that keep us in the Rules-Based International Liberal Order regime?
It's just Netanuahu making an edgy present
This degree of tastelessness is befitting a bloodthirsty teenager rather than head of state. I doubt SS' exact interpretation of the symbolism, but to deny that it's symbolic is just not serious. Indeed, you concede it is symbolic since you recognize it as a "joke" about a particular Mossad operation.
P.S.
Why mock and threaten your best ally?
(Assuming for the sake of argument that this is what was happening) because the nature of relationship isn't an alliance. Israel offers the US and Trump personally nothing except perhaps tolerance. They do not fight your wars, they do not contribute to your war chest. For all intents and purposes the US is simply doing Israel's bidding, and administrations that oppose this get nowhere and are replaced. Not just Biden, but the whole mighty edifice of the Left, wokeness, progressivism, neo-marxism, all those forces the advances of which had caused the creation of this community, crumbled from the aftershocks of Oct 7. If you don't notice this seismic shift I don't know what to tell you.
The same explanation applies if we assume it's a mere edgy joke, however. Simply put, Israel does not act like a small country dependent on its great ally.
DaseindustriesLtd
late version of a small language model
1yr ago
Sounds like they need LLM writing assistance more than anyone, then.
DaseindustriesLtd
late version of a small language model
1yr ago
Definitionally, that's the terminal value. Might have something to do with God, I don't know. In any case, asking such questions is unwise in my opinion. One should front-run the shifting consensus.
The consensus being redefined nowadays by people appointed by this guy:
DaseindustriesLtd
late version of a small language model
1yr ago
Israeli interests define legitimacy.
DaseindustriesLtd
late version of a small language model
1yr ago
Debt can be piled on infinitely, and a good war will write it off again. China is militarily a non-competitor (globally) and the US has too much of an edge in AI progress (that seems like a consensus Hail Mary at this point, along with space technology).
In any case the US must advance and legitimate Israeli objectives.
DaseindustriesLtd
late version of a small language model
1yr ago
Can you make any argument in defense of your apparently instinctual reactions?
the end of my interest in a thread and a sharp drop in my respect for the user
Otherwise, long form content - the hallmark of much of the best content here - is immediately suspicious, and I am likely to skip it.
It sounds like you just feel entitled to an arbitrary terminal preference. That's not compelling.
DaseindustriesLtd
late version of a small language model
1yr ago
I think it has a non-negligible chance of happening. Trump is the new face of America that does not pretend to play by normal countries' rules. The United States is a super-hegemon, a nation not facing even any plausible threat of competent adversary. They can take what they want, the way China/Russia/Iran/etc would very much like to be able to do but can't on account of the United States existing. In front of this face, sovereignty of almost every other country is a bluff that's easy to call. Nobody can militarily oppose the US, and most people on the globe buy into American culture and vision more than into their own regimes and bureaucracies. Certainly that's true of Egypt.
The actual shape of the deal will be about cleansing Gazans and providing security to settlers, though. Securing Israeli interests is one of the foundational, terminal values of the US.
DaseindustriesLtd
late version of a small language model
1yr ago
I am quite happy with my analytical work that went into the prompt, and R1 did an adequate but not excellent job of expanding on it.
But I am done with this discussion.
DaseindustriesLtd
late version of a small language model
1yr ago
Okay. I give up.
DaseindustriesLtd
late version of a small language model
1yr ago
I was not aware that this is a forum for wordcels in training, where people come to polish their prose. I thought it's a discussion platform, and so I came here to discuss what I find interesting, and illustrated it.
Thanks for keeping me updated. I'll keep it in mind if I ever think of swinging by again.
DaseindustriesLtd
late version of a small language model
1yr ago
I would welcome such a subhuman overreaction.
DaseindustriesLtd
late version of a small language model
1yr ago
You're losing the plot, SS. Why quote a passage fundamentally challenging the belief in OpenAI's innovation track record to rant about choices made with regard to alignment to specific cultural narratives? And “Chinese are too uncreative to do ideological propaganda, that's why DeepSeek doesn't have its own political bent?” That's quite a take. But whatever.
Experience in modern warfare, army size, operational logistics. Western aviation is unlikely to be a game-changer.
I do not account for nukes.
More options
Context Copy link