BANNED USER: temperamental dysregulation
DaseindustriesLtd
late version of a small language model
Tell me about it.
User ID: 745
Banned by: @Amadan
DaseindustriesLtd
late version of a small language model
3yr ago
If you mean the Ryodan thing, it's developing rapidly, so maybe some other time. I'll plug this post for the second part.
DaseindustriesLtd
late version of a small language model
3yr ago
Speaking of HBD, here's an interesting, very typical recent exchange on Twitter between a bunch of race realists and a certain Dialectical Biologist (self-identified Lewontin fan), with laymen chiming in too. The crux of the argument is, as always: can races have differences in average cognitive ability? The refutation, cutting aside fluff like «IQ tests have culture bias», is twofold: first, intelligence is always adaptive, and Africans also had to plan for the future in the ancestral environment (equivalently, «would have benefitted from long-term planning for e.g. droughts and... doing something about it). Second, races aren't even a thing, so the point is moot (human groups aren't species, aren't subspecies, aren't breeds, and so he just dismisses examples of cognitive differences within such subsets in non-human animals). E.g. here's an insane bit:
DB: 'Race realists' have never provided a coherent explanation for why/how different environments supposedly selected for high or low intelligence alleles during evolutionary history. High intelligence would be advantageous in all environments!
some guy: So is strength, speed, and height. Do those vary by race?
DB: Do you have reliable data showing those traits do actually vary by race?
some other guy: Pygmies exist
a third guy: How do you define different races?
DB: I don't, race is synonymous with subspecies, and humans don't have subspecies
So,
-
pygmies aren't a «race», seeing as «race»=subspecies =an invalid concept to begin with,
-
so there, by definition, cannot be reliable data relating to pygmies (or anyone) showing that strength, speed and height actually vary by race;
-
so the example of a specific lineage of humans being obviously short and weak can be safely ignored;
-
and this direction of attack on the core argument – inherent implausibility of differences in selection pressure on «more is always better traits» – remains guarded.
Obviously there are more layers to it: if this line of defense were somehow breached, he could concede that maxing out height isn't always better (even though I'd expect bigger pygmies to still be more successful in their environment) but intelligence is different, or some such. But he plays to maintain maximum optionality, not to score points in a rational debate, so even non-damning attacks must be deflected.
On the surface, such mental gymnastics, with rapid jumps between abstraction layers and revealed intelligence, seems to require explicit reasoning. I always wonder if these people understand what they're doing, or if they're RLHF'd into mentally circling the drain like ChatGPT is RLHF'd into incoherently preferring an H-bomb to the N-word, and are no more conscious than most folks imagine ChatGPT is. Or if there even is any substance to the idea that there could be difference. @Chrisprattalpharaptr, what say you?
But let's dig in. As evidence for races not being a thing, he shows that there is genetic continuity.
Population differences aren't perfectly continuous tho. Genetic distances between clusters are larger than what would be predicted from geographic distance
https://sci-hub.ru/https://doi.org/10.1371/journal.pgen.0010070
The boundaries also line up with geographic features like mountains + deserts + bodies of water etc
Dialectical Biologist responds with an insistence to accept authors' interpretation:
Yes I know. There is population structure as the papers you cite prove. This isn't evidence that the 4chan interpretation of the left PCA is correct. Also, I always find it astonishing how hereditarians cite papers by Rosenberg/Novembre and somehow miss points such as these:
As 0.0153 is not a large value of genetic distance, and because the addition of the B term produces only a modest increase in Rˆ2, the discontinuities that give rise to genetic clusters – as we have stated previously [3] – constitute a relatively small fraction of human genetic variation.
Our evidence for clustering should not be taken as evidence of our support of any particular concept of "biological race."
That's cute. Meanwhile, I cite papers for their results, not for their authors' hopes and dreams.
Couple things.
A) That's an R^2 statistic, which should always be thought of as more impressive than people intuit them to be (squared apples n such)
B) This is a test for incremental validity; the discontinuities are part of the reason why genetic distance increases with geographic distance (this controls for that; you're bragging that A has no effect on B after controlling for the effect of A). Similarly disappointing would be the effect of geographic distance on genetic distance when controlling for the discontinuities. The proper place for this is binary hypothesis testing, which this passes.
C) Red Winged Black Birds have recognized subspecies despite said subspecies having FST distances of only 0.9%, which the incremental validity here is singlehandedly stronger than.
https://sci-hub.ru/https://doi.org/10.1023/B:BIGI.0000012142.96374.b6
D) The meaningfulness of FST values differs among species (Remember, percent of variance; 100% of 10 is smaller than 10% of 1000). As it turns out, humans actually have an unusually-high amount of heterozygosity.
And here's what I was really leading to: a layman is indignant about him disagreeing with the interpretation:
Wait, your argument is that you understand the meaning and extensions of scientists' work better than they do? On what basis do you denigrate their research-based conclusions about the meaning of their own work, calling them hopes and dreams rather than research-driven positions?
[…] Ah, working scientific researchers, peer reviewers & journal editors don't understand simple things about statistics. It must be nice to be so much better educated, or naturally smarter, than active researchers. But, of course, human beings are like red winged black birds, so...
Basically, a layman's understanding of science is that it's a thing when special people do certain rituals and write some really hard stuff, and then their special credentials entitle them to recite a politically appropriate conclusion that receives Science Has Shown label. It's pure divination: priestly credentials + wordcel proof of work = Credible Truth that the Sovereign can appeal to when declaring a war of conquest, an early harvest or a tax hike.
But there's sad rationality, learned helplessness, underneath it all. A layman knows that he can be Eulered and confounded, so there is no point to examining whether those conclusions follow from the results, whether they even are conceptually in the same plane, and whether that interpretation is germane to the empirical issue that's being discussed. He can only hope The Experts are deserving of his trust.
(brought here by @official_techsupport's post in recent comments)
Studio hardware caps out at $8000; Mac Pro, at $50000. I imagine they will make some sort of Apple Silicon Mac Pro that stands above Studio, maybe with another doubling (or two doublings) of the next-generation Ultra. But usually their desktop is a very niche product and isn't refreshed as often. Also, its selling point is customizability that you can't (easily) have with these chips – the ability to get an insane core count, or memory on the scale of a decent Supermicro server, and still in the slick Apple package.
I think at this point it's more profitable for Apple to design and produce an all-around powerful compact SoC that radically improves their already-prestigious laptop line and crush the competition, rather than to fuck around with multi-part systems and market segmentation. They're capitalizing on their years of bespoke ARM engineering. It's everyone else who's making unjustifiably bad CPUs.
albeit more power-efficient
For some workloads. Also, TSMC's 5nm vs Samsung's 8nm helps.
DaseindustriesLtd
late version of a small language model
3yr ago
Not even Turing himself intended the Turing test to be a serious measure of capability, it is entirely a figment of journo/sci-fi writers imaginations. I think Bing passes it right now – sure, it's crazy and dumb sometimes, but humans are also crazy and dumb, and in much the same (although not identical) manner, with pigheaded obstinacy, gaslighting, deliberate obtuseness. And from the point of view of more credulous humans, ELIZA was passing it well enough already – so the idea that it's still an open problem is inherently elitist and subjective. Crucially, it's not testing what we want to test: a machine's sentience/intelligence/consciousness or whatever it is that we are interested in cannot be reducible to its ability to deceptively mimic a human or a very humanoid agent. It's both a simple task if solved with exploits, and a harder one than a mere human-level AGI if solved honestly.
A sentience that lives a single forward pass can have high superhuman «resolution», even if limited capability due to its meager context. Larger contexts, persistent «tape», training objectives and architectures emphasizing long-range coherence, clever prompts, other gimmicks can improve its external presentation, but I doubt they change much in terms of the peak cognitive power of what exists under the hood.
I've read dismissals of LLMs as glorified excel spreadsheets matching patterns of words together (Zvi, Gary Marcus, etc) and found them somewhat convincing
Well you've probably read some snippets from ChatGPT and Bing that are also delivered in an authoritative tone and cogently phrased, but turn out to be total bullshit under scrutiny. Marcus is more of a stochastic chatbot than a SoTA model, less amenable to persuasion, less interested in new evidence. I think we shouldn't worry too much about opinions of people who are outperformed by bots. Gwern's classic rant sums up this topic adequately.
however it seems like our understanding of intelligence and sentience are poor enough
I'd say our articulated understanding of what it means to «understand» something is laughable, and so we're making very little pop-philosophical progress in our discussion of how good our language models are. I want to write an effortpost on that, as well.
But plans and reality are different things.
DaseindustriesLtd
late version of a small language model
3yr ago
You absolutely could run them on a Macbook, and at decent speeds. Interactive decoding of the sort you'd need with a dialogue-oriented LLM is mainly bottlenecked by memory bandwidth needed to move weights between DRAM and registers, not by compute/processing cores. And Apple Silicon has insane theoretical bandwidth by CPU standards (M1 Pro has 200GB/s bandwidth, M1 Max has 400 and M1 Ultra, available on the Studio with 128Gb RAM, up to 800, vs. 90 for i9-13900K or 54 for the flagship Ryzen 9).
Nvidia cards with the same total memory would still be multiple times faster, though – and with the market full of ones used by miners, likely cheaper. Here's a good blog post on the topic, here's the list of most cost-effective ones. Note, however, that he doesn't worry about interactivity, so regards faster recent cards as more valuable. All things considered, gaymer RTX cards are so much better than A100s and such for many (though not all) tasks that Nvidia has a clause in its contract with datacenters prohibiting their use. 3090 (bandwidth 936 GB/s) is still a decent choice.
Power draw in the moment will be a bitch, of course.
DaseindustriesLtd
late version of a small language model
3yr ago
I don't think this is really comparable to the «Indian Cordyceps» so aptly described by Moldbugman.
Related take from Razib.
While I cannot know for certain, I think @HalloweenSnarry references the classical Prison Riddle, i.e. a shit test used as part of a «registration» rite for a new cell mate, popularized on Russian imageboards (this stuff was recently brought to the attention of Westerners by Galeev).
The riddle in question is a near-perfectly polished psychological attack against a relatively powerless newcomer in a honor culture setting; it's really pretty crude, but lazily googled translations are lacking, so here goes. «There are two chairs. On one, sharpened pikes. On the other, jerked [erect] cocks. On which one do you sit, on which do you put your mother?» As is common with riddles, it rhymes.
Supposedly, default passwords are:
-
«I'll take the sharpened pikes, cut down the jerked cocks, will sit myself and sit my mother».
-
«I'll sit on the pikes and sit my mother on my knees».
In principle, the universal counter «For what reason do you inquire?» (literally), or some blah like «We're fine with standing, thank you very much» are also valid, though I haven't had the opportunity to try it out.
There is a whole family of those riddles, sadly their charm is untranslatable. The most reductionist one is «Offer your ass vs. sell your mom?»
I like the trolley problem one:
You're on a train, chained to levers that can turn either left or right. There's a fork in the road ahead - your mother is tied to a pole on the right and your buds, ten of them, are on the left. Which way do you turn, who do you hit?
Answer: today's buds [could be] tomorrow's cops.
Russian (and broader Russophone) culture pays a lot of attention to the problem of choosing between terrible options and false dichotomies, captured in the saying «horse-radish ain't sweeter than radish». E.g. the Escobar Axiom of Choice (Escobar is a Ukrainian black metal character):
In any choice between only two mutually exclusive and opposite entities, both alternatives will be exceptional fucking shit.
Or in the original form:
«this [one] is fucking shit, and that one is fucking shit. Both fucking shits are such that I just fuck her mom's mouth».
Pelevin has developed this idea into a faux-dialectical method, e.g. in «Batman Apollo»:
– The Chinese Taoists, – he said, – had a similar notion, I will retell it in my own words. Struggling for hearts and minds, discourse workers constantly demand that people answer 'yes' or 'no'. All human thinking must flow, like an electric current, between these two poles. But in reality there are always three possible answers: "yes", "no" and "fuck you". When too many people begin to understand this, it means there is some wiggle in the skulls. In our culture, it has reached a critical point. It needs to be reduced drastically.
And earlier, in «P5: farewell songs of the political pygmies of pindostan»:
Ludwig Wittgenstein had claimed in his Tractatus Logico-Philosophicus that he had discovered a general form for describing the sentences of any language. In his view, this universal formula accommodates all possible signifier constructions – just as the infinite space of the universe accommodates all possible cosmic objects.
"That there is a general form of sentence," writes Wittgenstein, "is proven by the fact that there can be no sentence whose form cannot be foreseen (i.e., constructed). The general form of the sentence is: "The issue is so und so" ("Es verhält sich so und so").
However, philologist Alexander Sirind, an associate professor at the Irkutsk Pedagogical Institute, recently managed to disprove the famous formula, by giving an example of a sentence that goes beyond the all-encompassing paradigm outlined by the Austrian philosopher. It goes like this: "Fuck you, Wittgenstein".
But as for the CCP vs the Globohomo, I think another piece of imageboard fancy is more relevant – the 4chan «freedom is best, and hard choices» or maybe the Ben Garrison remake «the march of tyranny». If you don't have negotiating power except in the form of defecting in protest, and your defection threshold is only reached after both tyrants have curtailed freedoms of their subjects, just to a different extent (and that's still a better case than what we have here) – eventually both tyrants converge to absolute dominance and all subjects are maximally debased.
DaseindustriesLtd
late version of a small language model
3yr ago
He surely would. Sorry, was a bit distracted, see edit:
This is literally a racket with clear finite conditions and demands, coordinated by a high-profile activist group, with a specific man, called Jonathan Greenblatt, calling the shots, as they say. They have invested a lot of effort into making it appear as some decentralized grassroots movement they have merely volunteered to champion and which would have meaningfully existed regardless, but this is just a practical expression of what Yarvin talks about here:
Bluntly, I do not believe that leaders, coordinators, instigators, Machiavellians do not matter and do not determine how the vague popular sentiment and antagonism swells and turns. In some galaxy-brained Civilizational Big Picture it may be different, but on the timescale of real politics, at least, wins and losses are largely determined by the quality of commanding corps and their ability to not be handicapped, distracted and dispersed.
To think otherwise is supremely democratic, ironically. Yarvin consistently downplays the relevance of specific decisionmakers, unusual capabilities and intelligence (see his rhetoric on AGI). Well, I've already explained how I interpret this impulse.
Or it might not - after all, you can call the villification of Schwab a lot of things, but effective probably isn't one of them
To the extent that Schwab is not a figurehead, it could be effective. I just don't think Schwab is who he is alleged to be by right-wing conspiracy nuts. Greenblatt's authority over his organization is not a conspiracy theory but a fact of life, a fact that upsets even Jews who feel he's too focused on culture war bullshit they didn't subscribe to, compare him negatively to Abe Foxman, and it is plausible that him being replaced by a straightforward Bibi-style Zionist, as they'd prefer, would change matters for the better (or for the worse). Schwab, like @2rafa says, is for the most part just an old man, a normie European economist who likes to run conferences – he does not even peddle ideas right wingers attribute to him, nor does he have the power to compel others to act on his actual views. Actually the degree of Schwab demonization should be suspicious in itself – it's almost like he's a lightning rod to distract enemies of the forming political consensus (which is, incidentally, not «globalist» any more) from its high-agency advocates. See the previous point.
I have no deep up-to-date knowledge of this subject, particularly none applicable in the West. Read Gwern on this topic and dark markets more generally, if you want. That said, if I were interested in microdosing, I'd have
-
looked for shops with high reputation and indication of sourcing from guys who have the technical chops, on Hydra or whatever is the current mainstream platform. This is easy in major Russian cities due to the availability of poor, high-IQ chemists and badly supervised professional equipment, so e.g. phenethylamines are a solved problem. I suppose in the West it should be easier still. For LSD, there can't be that many reputable producers, but I'd guess the competitive pressure has fixed global supply, with consistently glowing reviews. (I remember my junkie friend talking about voidgoblin of something; keep such friends around if only for darkmarket questions, this isn't like Amazon where you can kind of afford atomization).
-
bought hardware for dosing and storing sensitive chemicals, such as precision scales, amber glass vials, I don't know.
-
thought about diluting the dosage appropriately to increase the precision of distributing it. E.g. assuming away some inconvenient complications that I advise people look into themselves, LSD should be soluble in distilled water and pure ethanol (for preservation against microorganisms), and it's a heck of a lot easier to measure milliliters than micrograms. For something like 2C-B, decent scales from Aliexpress should do it but you can also mix it with some inert substance like sugar.
But ultimately, what you need is not objective measurement but reproducibility of trials. Your individual sensitivity determines the dose you'd desire to administer (provided microdosing does anything of value) – and you will have to approximate it by trial and error.
Re: crime and law: lol. lmao. Of course it is a hazard. Knowledge of this will also make you paranoid under the influence. The risk can be minimized but not eliminated.
Pinging @JhanicManifold.
Supply-and-demand is nice but the observation of hierarchical organizations that can create and channel demand should temper our expectations for its predictive power. The word «mercenary» is ambiguous too: mercenary attitude is just opportunism, but mercenaries have clients, don't they? There are many mercenary characters in Wagner; do you suppose those convicts would have had a particular effect on the war without Putin's bidding and Prigozhin's command? And likewise, there was a de facto client in the actual, concrete case of journalists and advertisers bullying Facebook. Yarvin says shit like:
The more Facebook censors, the clearer it becomes that Facebook is an accessory to murder and profits routinely from hate. Corporations are used to dealing with _rational_powers like the state, for which compliance decreases pressure. They are not at all used to dealing with the psycho type of power, for which compliance increases pressure—like Tony Soprano busting out a sporting-goods store.
So the victim creates the bully. The fact that Facebook can be bullied makes people—both journalists and their readers—_want_ to bully Facebook. Facebook’s _power leak_produces a kind of oasis of power—water in the desert. The water causes the palm trees; the palm trees don’t cause the water.
What we see here is a case of spontaneous coordination. The palms were not ordered to grow around the leakage pond; the bears were not invited to a bear party. When we try to understand what happened, we must start from two axioms: that there is order here, and that there is no center to the order.
To be clear, Mr. Zuckerberg has not yet approached the type of meaningful action that we want to see. The issue is not that Facebook just lags competitors in working systemically to address hate and bigotry on their platform. To use a favorite term of Facebook’s leadership, Facebook’s attitude towards seriously addressing how their algorithms push hate, violent conspiracy theories, and disinformation is transparently “inauthentic.” Mr. Zuckerberg treats meetings and dialogue as outcomes. He puts more effort into obfuscation, lobbying, and distribution of misleading talking points than seriously addressing the deadly consequences of his choice to profit from hate. In the words of one of their own engineers who resigned this month over leadership’s unwillingness to take action on problematic content, “Facebook is hurting people at scale.” Society cannot afford the Facebook status quo. [...]
This movement will not go away until Facebook makes the reasonable changes that society wants. The ad pause in July was not a full campaign – it was a warning shot across Facebook’s bow. This movement only will get bigger and broader until Facebook takes the common-sense steps necessary to mitigate the damage it causes. And it has spurred additional constituencies who also are demanding change. We saw this demonstrated in full force yesterday in Congress where legislators forced Mark Zuckerberg to testify and held him accountable for Facebook’s failures. And we expect more constituencies will emerge in the coming weeks as this movement gains even more momentum.
Now, we know that change will not happen overnight, and we remain willing to engage with Facebook when they are prepared to commit to a public timeframe and substantive action relative to our very straightforward demands.
Mark Zuckerberg, the ball is in your court.
This is literally a racket with clear finite conditions and demands, coordinated by a high-profile activist group, with a specific man, called Jonathan Greenblatt, calling the shots, as they say. They have invested a lot of effort into making it appear as some decentralized grassroots movement they have merely volunteered to champion and which would would have meaningfully existed regardless, but this is just a practical expression of what Yarvin talks about here:
The thought of studying itself is inherently foreign to power. Power does not want to know itself. The most powerful powers do not even think of their power as power. If power does know itself, it keeps that knowledge to itself; but mostly, it really does believe its own official myths. The real O’Brien is a rare figure.
Power is indeed decentralized and cognitively compartmentalized in many interesting and important ways. But if we allow speculations on Yarvin's level, it is perfectly parsimonious to not stop here and say he's also a facet of the purported Cathedral: by equivocating and talking in the abstract about concrete clashes, distorting their timelines and incentives, peddling warped axioms and very articulately persuading potential enemies of the power to not look at its crucial high-agency nodes, he maintains his good standing with other... elves.
But I'm no hobbit. I hail from Mordor.
DaseindustriesLtd
late version of a small language model
3yr ago
StopHateForProfit was only spontaneous in Yarvinverse. There's a lot of posts about it back on the sub.
DaseindustriesLtd
late version of a small language model
3yr ago
This kind of manifesto-posting is not desirable
It's not even a manifesto. The last time it was more well-formed but also fell short of our classical manifestoposters. It's «here are some reasons I find compelling to think that the Chinese are, essentially, yucky emotionally stunted robots. Amirite?» He suggested some racial struggle, but what's the struggle? Chinese babies are significantly more chill than other babies; Chinese adults tend to wear masks, excel at technical competence and fail at entering the PMC; a hundred years ago, Mainland Chinese elite women had their feet bound. Okay, I personally buy all this and much more.
Where's the thesis and its development? What is supposed to be or not be litigated, exactly? That a sovereign Chinese state is inherently a threat to Western values or something? This doesn't follow from the provided evidence, such as there is, and isn't even articulated.
Maybe I just lack the context of the Yellow Menace discourse and it's assumed to be self-evident the moment Chinese differences are established. I can certainly see how an intuitive antipathy for a racial Other can inform policies. But this is supposed to be a place for rational-ish discussion. You need to spell this stuff out.
There is asian overrepresentation in every field involving intelligence and a bamboo ceiling in every subfield requiring a personality
Do you suppose high-caste Indians and Jews have incredible personalities to go along with their intelligence, in the sense that there's «more» of a person there, or a better person? Might this word be deserving of some… scrutiny? This autistic Chinese American guy has an opinion to share:
You, the reader, have probably noticed that up to now, we’ve focused mostly on brains and technical ability. Yes, they are essential, but personality characteristics (both individual and collective) and “soft skills” also matter, especially if one wants to rise to a leadership position. From my personal observation, Indians are, in general, very good at projecting confidence and assertiveness from the way the talk and present themselves, much better than Chinese are, at least in the American cultural context, even when you discount the language barrier Chinese face relative to Indians. I’m talking not only about how one says things in terms of word choice, but the vocal tone and body language behind it. Sure, you can disdain this as superficial, but it matters. Perception matters as much, and in some cases, more, than substance. There is also that Indians seem to have a stronger network and help each out more in the career world. Collective intelligence or ethnic nepotism, you be the judge.
I have stories to tell on this. First of all, I remember vividly how when I interned at the same place as an Indian schoolmate, he was the only one who scheduled, successfully in a few cases, coffee meetings with executives, as an intern (!!!!!), when it never would have occurred to me, or probably almost everyone else except him, to even try. One can sort of link this to collective intelligence, in that it is an indicator of discernment with regard to who matters (the executives) and who doesn’t (the engineer worker bees) within the political organization. And needless to say, you rise up in the organization by aligning yourself with the people who matter. Yes, my telling a full-time engineer this was met largely with a response in the likes of, “He knows who matters and who doesn’t. And even if he completely fucks up, he has nothing to lose, he’s only a 2nd year college intern. In any case, he gets good practice interacting with people who matter.” There is also that multiple people I know have complained about blatant Indian favoritism in interviews in the likes of what is described in this Quora answer. Yes, others have told me that when Indians interview other Indians, the bar is much lower. It’s not just in interviews. Another guy told me about how he once worked for a company that turned into ruins after Indian managers protected some Indian fuckups from getting fired. Personally, I have seen a case of Indians getting promoted way faster than those of other ethnic groups on a big team with an Indian director. So sometimes, I ask myself the verboten. Could it be that Indians really are far higher ranked in tech companies than their ability and contribution, because they are much more self-promoting and collectively nepotistic than those of other groups? Moreover, could it be that many people secretly think and resent this but are too afraid to say out of fear of being publicly vilified for “being racist” and having their careers ruined from alienating a national group increasingly powerful in corporate America? And that gradually, other groups, as they awake to the rigging of the game and get past, reluctantly, their moral objections, will quietly do the same, transforming tech companies and the American workplace at large into literal prison gangs contend, destroying whatever is left of the ideal of meritocracy and fair play in this country, ever more mired in identity politics?
I remember Lynn (probably) writing that black people, too, tend to have «winner personalities», just of a slightly more combative mold, which makes teachers intuitively assume that black students must be smarter than what those ignorant, culturally biased formal tests might indicate: always speaking up in class, not ashamed to ask questions, proud, undeterred by criticism or low-class background… basically model grinning Americans from cartoons and stock photos, who don't even need no Tony Robbins to coach them into success. The other side of that is surprising lack of real success, and probably stuff like this too. The Chinese are more known for videos of this kind. As you say, your struggle with China is racial. So, which way, Western man?
Aside from suggesting that the Chinese aren't full-fledged human beings because something something not emotionally reactive enough, do you have an idea of a perfect American? Less like the Chinese, more like Indians, Jews, Blacks? Or do you suppose that Whites are the only group that's properly balanced? I've criticized them the last time and could go on with mockery, but would you mind making a positive case?
DaseindustriesLtd
late version of a small language model
3yr ago
Apologies for the abrasive tone: I take issue with your method and authorities, but would prefer you not to take it as a personal attack.
Funny that you talk about peeking under the hood, but later refer to Hofstadter. I'm sick of hearing about this guy – and cringe at the whole little adoring culture that nerds and «hackers» have built around his books. GEB and strange loops, this intellectual isomorphism of autofellatio, self-referential kabbalistic speculation detached from all non-synthetic evidence and loudly, proudly spinning its wheels in the air. «Dude, imagine thinking about… thinking! Isn't that, like, what programmers do? Woah…» It is pretty sad when a decently smart brain belongs to a man who happens to build a brand out of a single shower thought and gets locked by incentives into inflating it endlessly! Even worse when others mistake that for an epiphany, generations of poorly socialized kids looking for the promised deeper meaning, and establishment journalists respectfully asking the matured stoner for his Expert Input. (Then again, I may simply be envious).
Last I've seen Doug speak of machine learning, it was July 2022, and he was smug because he tricked GPT-3 into hallucinations (The Economist) :
I would call gpt-3’s answers not just clueless but cluelessly clueless, meaning that gpt-3 has no idea that it has no idea about what it is saying. There are no concepts behind the gpt-3 scenes; rather, there’s just an unimaginably huge amount of absorbed text upon which it draws to produce answers. But since it had no input text about, say, dropping things onto the Andromeda galaxy (an idea that clearly makes no sense), the system just starts babbling randomly—but it has no sense that its random babbling is random babbling. Much the same could be said for how it reacts to the absurd notion of transporting Egypt (for the second time) across the Golden Gate Bridge, or the idea of mile-high vases.
This is not to say that a combination of neural-net architectures that involve visual and auditory perception, physical actions in the world, language and so forth, might not eventually be able to formulate genuinely flexible concepts and recognise absurd inputs for what they are. But that still wouldn’t amount to consciousness. For consciousness to emerge would require that the system come to know itself, in the sense of being very familiar with its own behaviour, its own predilections, its own strengths, its own weaknesses and more. It would require the system to know itself as well as you or I know ourselves. That’s what I’ve called a “strange loop” in the past, and it’s still a long way off.
How far off? I don’t know. My record for predicting the future isn’t particularly impressive, so I wouldn’t care to go out on a limb. We’re at least decades away from such a stage, perhaps more. But please don’t hold me to this, since the world is changing faster than I ever expected it to.
Narrator's voice: «they were 4 months away from ChatGPT». Today, pretty much the same text-only GPT-3, just finetuned in a kinda clever way for chat mode, can not only recognize absurd inputs, but also explain the difference between itself and the previous version, better than Hofstadter can understand. This was done on top of the previous InstructGPT tuning, also misrepresented by Experts On AI with what is basically a tech-illiterate boomer's conspiracy theory:
Artificial intelligence is an oxymoron. Despite all the incredible things computers can do, they are still not intelligent in any meaningful sense of the word.
InstructGPT is then further fine-tuned on a dataset labeled by human labelers. The labelers comprise a team of about 40 contractors whom we hired through Upwork and ScaleAI.
OpenAI evidently employs 40 humans to clean up GPT-3’s answers manually because GPT-3 does not know anything about the real world.
I told one of my sons that the hand labelers would probably fix these glitches soon. Sure enough, I tried the same questions the next day, March 19, and found that the answers had indeed been cleaned up: ….
Gary: Can I use random numbers to predict presidential elections?
GPT-3, March 18: There is no definitive answer to this question. It depends on a variety of factors, including the accuracy of the polling data and the margin of error.
GPT-3, March 19: No, you cannot use random numbers to predict presidential elections.
The labelers will probably clean this up by tomorrow, but it doesn’t change the fact that so-called AI algorithms still do not understand the world and consequently cannot be relied upon for sensible predictions or advice.
Today we know that LLMs have what amounts to concepts. Today, people have forgotten what they had expected of the future, and this Sci-Fi reality feels to them like business as usual. It is not.
Every little hiccup of AI, from hallucinations to poor arithmetic, its critics put into the spotlight and explain by there not being any real intelligence under the hood, the sort they have. The obvious intellectual capacity of LLMs demonstrated by e.g. in-context learning is handwaved away as triviality. Now, like Boretti says, «frames or symbols or logic or some other sad abstraction completely absent from real brains» – now implementing that would be a «big theoretical breakthrough». We don't know if any of that exists in minds in any substantial non-metaphorical sense, or is even rigorously imaginable, but some wordcels made nice careers out of pontificating on those subjects. Naturally, if they can be formalized, it wouldn't be much of an engineering task to add them to current ML – the problem is, such reification of schemes only makes things worse. The actual conceptual repertoire developed by humble engineers and researchers over decades of their quest is much more elegant and expressive, and more deserving of attention today.
Do you really think that the idea of «predict next word in a sentence» provides sufficient insight about the under-the-hood intelligence of LLMs, when it is trivial to change the training objective to blank-filling, and RLHF guarantees that there exists no real dataset for which the «predicted» – or actually, chosen – token is in fact the most likely one in that context?
Or that the process of self-attention , actually undergirding those «predictions», is not «reasoning about reasoning» (because it cannot attend to… itself, the way you can attend to patterns of neuron spikes, presumably?
Or that recurrence is hard to tack onto transformers? (or for that matter specialized cognitive tools, multimodality etc.?)
And so on and so forth. But ultimately after years and years of falsified forecasts and blatant displays of ignorance by skeptics, it is time to ask oneself: isn't this parlor trick of a stochastic parrot impressive enough to deserve more respect, at least, than gimmicks of our fraudulent public intellectuals? It does, after all, make more sense when talking, and is clearly more able to grapple with new evidence.
@2rafa reasonably observes that human intelligence may be not so different from next word prediction. Indeed, if I close this tab (in Obsidian) and return to it in half an hour, I may forget where I was going with this rant and start with the most likely next word; and struggling for words when in an intense conversation is an easy way to see how their statistical probabilities affect reasoning (maybe I'm projecting my meta-awareness, lol). But even if the substrate is incompatible: why do we think our one is better? Why do we think it supports «real reasoning» in a way that mutiplying matrices, estimating token likelihood, or any other level of abstraction for LLM internals does not?
It is not obvious that the human brain is anywhere near optimal for producing intelligent writing, or for much of anything except being itself. We didn't evolve to be general-purpose thinkers, we are just monkeys who had our brains scaled up under selective pressures in some limited range of environments, with stupid hacks like the phonological loop and obsession with agency. Obviously LLMs are not optimal either (we are only pursuing them out of convenience), but they might still be better at producing our own text.
A plane doesn't flap its wings, but it definitely flies – and even though it's less efficient per unit of mass, in the absolute sense it does something no bird could. We do not understand birds well enough to replicate them from scratch, nor do we need to. Birds could never achieve enough for a truly general flight, «move anything across the Earth» kind.
We, too, aren't enough for truly general intelligence. It remains to be shown that LLMs aren't better fit for that purpose.
I haven't read your links yet; this is some complex stuff with lots of plausible deniability. One of the reasons I'm grateful for my Russian background is precisely that everything is so much cruder than in the West, even though directionally similar, with comparable demographics, history and memes. Contrasting those two worlds, I think, lets one learn more about both.
In this case, I've been looking into «PMC Ryodan», an utterly bizarre… thing that has just happened. You can read the story on Bellingcat (actually a decent source) and whatever this outlet cited by Yahoo news is. Since the motto of the day is «believe women Ukraine», the story of security-minded Ukraininan boomers is cited without comment:
Tymoshko believes the youngsters were gathered in the city by Russian security services through “manipulation and deception, and they should have started a fight so the Russian TV could use it.”
Besides the violence, PMC Ryodan is also notorious for its intolerance of migrants and the Muslim population living in Russia’s Caucasus region.
PMC Ryodan, the acronym short for “Private Military Company,” reportedly took its name from a group of ruthless bandits in the popular anime and manga series “Hunter x Hunter,” created by Yoshihiro Togashi
As one could expect, Russian boomers accuse Ryodan of being a Ukrainian psyop, and zoomers on both sides are having the time of their lives (except those who have to fight the actual war).
But this isn't the point. The point is: it isn't really something that is happening for real. There is no PMC Ryodan, outside of the Hunter X Hunter universe at least – or rather, there wasn't. Just some weeb teens who bought HxH merch, and a couple online groups that used the PMC meme, and kids who were assaulted by gopniks in a food court. Everything else was spun out of this seed – including, unfortunately, the minor campaign by actual thugs to hunt weebs, in a mockery of a turf war.
…Actually, this isn't the point either, as you may expect. While checking out the Ryodan stuff on the Russian Reddit analog (Pikabu), I've seen this story as a comment (abridged):
I have wanted to tell this story for a long time, for it is damning and instructive in terms of the perception of information, the value of eyewitness accounts, the spread of rumours and the ability to find the essence based on media coverage. I just wanted at least 200-500 people to read it, and not the way I usually have 20-50 (apparently my rating is not enough), and then I might lose interest… or get sclerosis… or Pikabu may get canceled…
So... it was the distant year 2000, I was a young detective in one of the city's district offices, it was a warm September day, I'm busy with detection and prevention and stuff... and a message by pager: «Go to the District Police Department XXX, there's an attack, pogrom and a generally crazy mess, you will assist local operatives».
So here's the deal:
There's a school in their neighborhood, and not just any school, but a Jewish one. That day it got attacked by extremists/Nazis/Fascists and such. They broke down the doors, threw desks and chairs, beat everyone with sticks and chains, drew a Star of David on the forehead of an elderly teacher with a marker, and then spat on it. Children and Jews are terrified, demanding security, defense and retribution. And because the governor, the police chief and the prosecutor are on the chase, and soon the UN will fly in, Interpol will be involved… and if these scum aren't caught and punished, it will be an international scandal and the whole police department will get circumcised.
Of course I asked them if I could go with the riot police to the safe houses and places where the rabble gathers, so to speak... and procure the enemies of the Jewish people there?
You see, the answer was, in fact, that there was no need to look for anybody, because that's not exactly how it happened. Two local kids looked into the class where that Most Victimized Teacher was giving a lesson, and asked out Izya... let it be Сockpoonberg... to go have a beer around the corner together, or for a loan, it's not clear any more. The teacher, of course, was against such a breakdown of discipline (or the issuance of loans without guarantors) and kicked them out... They said «ew, Jewish snoot», spat (but it's not exactly clear where) and some shit happened while pulling the door at each other, resulting in the glass breaking (but it's also unclear, maybe it was slammed, maybe even kicked). So they ran away at once. The teacher could not tolerate the humiliation and immediately told the school principal, who called in all his connections, and then, as Harrison said: «…a bird dropping, falling on a snowy mountainside, rolls down, accretes snow...» And so now we have a high-profile case that will never be solved, despite the demands of all the controllers to solve it as soon as possible. Why? Well, how do we solve it now? Either everybody has to admit that they blatantly inflated the event and gave false evidence, or make terrorists out of those guys and put them in jail for 10 years (remember, that was 2000, now I think option B would be picked in a heartbeat). That's why we are sitting there writing/searching.
Then the highway patrolmen bring in a guy, early 20s, who got busted a couple of years ago in some shady companies for either fighting or petty hooliganism. Obviously no one was going to tie him to this case, but he could provide the most valuable information on what kind of Nazis our city has come to harbor and where to find them. After a long "tell me something about the wonderful world of extremism" conversation, the guy told me the following:
Back then he fell in with some skinheads… or Punks, among whom there were two-three really colourful characters, some «Awl, Bullet and Loaf», but that's not certain: one even had white laces on his combat boots [in 90s-00s Russia it used to be popular with Neo-Nazis/WNs, theoretically only allowed after you've killed a non-White or distinguished yourself in a fight; really there was no enforcement], the second had leather trousers, and the third had a skull earring (possibly even human-skull-shaped!!!). He hung out with them once after a concert, got wasted and started quarrelling with someone, after which they got nicked. Since then he''s no longer friends with any skulls and white shoelaces, doesn't go to concerts and only ever drinks kefir.
That's it, think how much paperwork you can put into it! All-points bulletins went around the region, such and such – Awl, Bullet and Loaf wanted for implication in an assault, all wearing white laces, leather trousers and skulls on all sides, identify, apprehend, orient… I spent like a whole another month on these orientations, wrote all kinds of papers about trying my best to find these scoundrels, and even one of the detainees told me that he saw a man wearing white shoes with black laces, but can't recall where, because was freaking out from withdrawal. What am I saying… all the police, Minors Affairs and district officers were writing volumes of reports, and the police department was supervising them… cases were not going to blow themselves up. I honestly don't know how it ended, but judging by the fact that I don't know, it ended with nothing, as was the plan. The Jewish community (its individual representatives) waited in vain for their retribution; the high-profile case became lowercase, the journos guzzled away their fees, and now somewhere in the depths of the Department of Internal Affairs there rests a massive case about a terrible pogrom in a school.
But now, when I am told, «We have studied the declassified archives and, in fact, the truth is this, 100%» – I smile condescendingly.
Yes, that's where they usually ask for proofs... well, enjoy:
From the above:
…Pupils hid under their desks and ran to their teachers, who tried to talk sense into the young Nazis: «Guys, please go away, there are children here». One of the female teachers said:
– I lost my bearings. The scumbags should have been kicked out, but I was afraid that if I raised my voice at them or said something rude, they would disfigure the children.
The pogrom lasted about five minutes. Then the command sounded: «Stand up!» The rioters lined up, raised their hands in the Nazi salute and shouted, «Sieg Heil!», and then leisurely walked out of the destroyed school. One of them turned around and shouted: «We'll be back!»
– By happy coincidence, no one got physically hurt, but the psychological shock was very strong, – says Leonid Reznikov, head of the Jewish community. – Now parents are afraid to send their children to Sunday school, fearing for their lives. The way the rioters acted showed that they had been preparing for this operation beforehand. There are several distinctly anti-Semitic organizations in Ryazan. But the local authorities are inactive. Next time a pogrom could end in blood» […]
Investigator Irina Novikova refused to talk to Kommersant about anything to do with the pogrom at the Jewish Sunday school in the Soviet district of Ryazan.
The day after the pogrom, a leaflet appeared on the wall of the school with a swastika and the text: «Death to the Jews! We will be back».
One is free to choose what to believe.
DaseindustriesLtd
late version of a small language model
3yr ago
I'm not sure how much of this to believe; if it is dumber, that has to be explained somehow.
In my experience, the subjective generic text completion (and iirc perplexity) – is worse, even if most downstream tasks end up being better, and while the mean of sensible instruction-following is higher for RLHF-version, the variance is much lower, so you struggle to get ChatGPT out of its midwit persona in a way that isn't true for davinci (though this may be damage from arbitrary preferences and political bias, rather than the inherent damage of the technique). Opt-in will probably be incentivized.
Even at int4 (which is still dumber in the absolute sense, just more efficient per byte used) a 175B model uses >2 A100-40gb, and if they mix layers it's more. And for a low-latency interactive app, the utilization of 50% sounds rosy – Google with PaLM barely broke 40% in their paper.
Assuming they use 16 bit floating point
We've already assumed int8/4 though, but in any case the bottleneck for interactive use is memory bandwidth, not flops. We'd need more details.
I know there exist (at least on paper) techniques for making inference way, way cheaper, probably even cheaper than you suggest, with minor loss in quality – just not sure they have deployed them yet.
I just plain disagree, and feel compelled to build a thin client product on top of the API to prove it
Here's some free lunch: take LLaMA-65B and rent some GPUs, it'll be 2x cheaper using all the same tricks and allegedly about as good.
There are many jobs that are mostly low-skill word manipulation
I wonder if we will discover that there are many jobs which are, in fact, net negative for the economy only when people doing them get automated away and their sinecures lose lobbying power .
I suspect it, too, is a gimped/lossily accelerated version, with the full one to be released under the «DV» brand as part of their Foundry product. I may be wrong, however, and ultimately this PR-driven nomenclature is making less and less sense, they have trained like a dozen major models at this point, and more finetunes. Same story with Google's PaLM.
More importantly, quantity has a quality all of its own. DV offers a context window of 32k tokens. If the model can be cheaply ran with such enormous contexts, this will more than compensate for some intellectual deficiency in the context-sparse text-prediction mode. You have seen effects of StableDiffusion mega-detailed prompting, and of prompt prefixes that amount to a single page – now imagine 20 pages of precise specification, few-shot examples, chain-of-thought induction, explicit caveats for every failure mode discovered in testing; basically a full employee's manual. Writing and testing these megaprompts may become a new job for ex-middle managers who have suddenly transformed into leaf nodes of the org chart – for a short while.
DaseindustriesLtd
late version of a small language model
3yr ago
Not Fun thread material, fit for a separate post.
This week, the company released their ChatGPT API which uses their "gpt-3.5-turbo" model, apparently the best one yet, for the price of $0.002/1000 tokens
Well, as you quote, they imply it's not the best one yet, at least in «some use cases», and some experiments show it indeed is intellectually inferior to the most recent naive GPT-3 version; maybe this is inherent to ChatGPT model we see in the demo, because it is known that RLHF can give models the computational equivalent of brain damage as far as benchmarks are concerned, maybe it's something specific for the new one. I am not sure how they've achieved the price cut (though my intuition is that this is cynical undercutting at a loss to nip competition in the bud and keep the data flywheel accelerating) – perhaps it's smaller (trained from scratch, distilled, etc.), or aggressively quantized, or more greedily sampled, or they've somehow increased the batch size into high thousands, whatever. In any case, Altman is a great showman and this may not be the revolution it seems like at the moment. Do we really need an endless stupidity generator with short context? Most people who are on this level (and I do think we can now meaningfully say that some people are on this level) around ChatGPT) aren't exactly making money with their smarts. It's nice for employers to automate even more drudgery, of course.
But in principle, yes, optimizations and next-gen models with more novel architectures than Vawani!Transformer will definitely allow even cheaper inference (I think GPT-4 under the hood of Bing already is that).
DaseindustriesLtd
late version of a small language model
3yr ago
I was a bit busy today (while my history may lead some to think otherwise, I actually have things to do other than post here).
My impression is that the requirement for «some effort» is reasonable and must be enforced with the usual stick; but the toxicity inherent in arguing about it, chastising people who make use of an easy opportunity, and comparing them to others also imposes some costs. The leak, potentially massive news, has been available for nearly a whole day, and I only learned of it something like 6 hours ago – 2 hours prior to OP making his move. This was almost an inevitability.
Like already answered, this is the number of parameters. A parameter is the same thing as a weight, a unit loosely inspired by the synapse in biological systems like ourselves: a coefficient that is adjusted during training to reduce the predictive error, maximize reward or however else the objective function is defined for the purpose of a given project.
You can consider the number of parameters to be a measure of a neural network's expressivity: theoretically, the more parameters there are, the more algorithms, or more complex ones, can be learned/approximated by the model (this is a nice elegant illustration of the sense in which a neural network learns to represent an algorithm). But in practice, for now it seems that most models, and virtually all models released prior to Google's Chinchilla, are grossly overparametrized: a smaller network trained in a reasonable way on the same amount of data learns more or less the same skills, and a smaller model trained for longer learns qualitatively more, in that it actually reaches the underlying algorithms that allow it to find solutions in the general case, and doesn't just memorize superficial patterns or even raw data itself.* In this case, LLaMA-13B (13 billion parameters) is allegedly equal in benchmark performance/apparent "intelligence" to GPT-3-175B, so it's more parameter-efficient by a factor of 13,46, and also vastly more efficient in terms of training expense. The main secret is that it was exposed to 1 trillion tokens (a character group that's basically equivalent to a short word, see here), whereas GPT-3 only saw 500 billion. (It must be added that the average LLaMA token is shorter, because it uses character-level tokenization for numbers, so it should also have better arithmetic). The biggest LLaMA is trained on 1.4T tokens like Chinchilla-70B (with the same caveat about tokenizing numbers) and, for some not so trivial reasons, is slightly better still.
Aside from the total number, what matters is parameter precision. Models are usually distributed with fp32 weights. As Elon Musk notes, int8 (1 byte per parameter) is fine for inference. @ThenElection may be wrong here, I think 7B and even 13B will run just fine – after some tuning by nice anons, of course – on recent Apple Silicon Macbooks, with even 33B possible on top-of-the-line 64Gb version** (curiously, in one benchmark, 33B model is superior to the 65B one).
See @Porean's experimental results here and the recent AAQC winner @TransgenicSolution's related note here.
*That said, super-large models still seem to have unique emergent capabilities, though as we proceed with training Chinchilla-proportioned models, fewer and fewer such capabilities remain. Before UL2-20B, the consensus was that you need like 60B or 100+ to get advantages from chain-of-thought prompting.
** tfw no 64B M3 macbook to run your personal genie
Edits: typos
DaseindustriesLtd
late version of a small language model
3yr ago
Oh well, another opportunity for an effortpost lost.
Leaker Anon is the true Prometheus of our age, no doubt. If benchmarks are to be believed (and I emphatically do not believe Facebook's results until they are validated externally) – this is a near-human-level intelligence, as far as text is concerned. The smaller models will run trivially on consumer hardware, while promising GPT-3 level performance; the bigger one is allegedly competitive with PaLM.
We shall see.
DaseindustriesLtd
late version of a small language model
3yr ago
DaseindustriesLtd
late version of a small language model
3yr ago
I am not sure how the notion of shit tests (which has merit, but which PUA/Jim types already overuse, to the point I think can be legitimately identified with rape culture) translates to the level of more or less abstract society-wise policy demands. For example, women overwhelmingly support «more spending on underperforming schools» or some bullshit. Is this shit-testing school statistics buffs or, say, just misguided motherly instinct and wishful thinking (that I talk about)? I think we don't even need such folk categories; normal differential psychology explains enough.
Not sure how you'll take this but I've enjoyed this autistic rant that explains «models» by preferences of predominantly gay modeliers rather than median men or organic looks hierarchy of women.
Edit: I may be biased here. One of the most miserable women I know is a short, voluptuous and fairly inept girl who very reasonably guessed she could make an easy living with her looks (a few mutual friends have rated her 10/10, I'd say 9 is about right), but tried to achieve this through traditional modeling – which completely destroyed her self-esteem through comparisons with lanky scarecrows sporting chiseled cheekbones. Honestly, everyone likes to hate on onlyfans, but in her case it would have been better.
More options
Context Copy link