site banner
naraburns  nihil supernum  1yr ago (text post)   3539 thread views

Small-Scale Question Sunday for January 29, 2023

Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?

This is your opportunity to ask questions. No question too simple or too silly.

Culture war topics are accepted, and proposals for a better intro post are appreciated.

4
Top Bottom New Old Comments
Jump in the discussion.

No email address required.

Inference of LLMs isn't that much of a problem, because batch size scales rather cheaply, and there are tricks to scale it even cheaper at production volume. OpenAI is still burning through millions in a month (we aren't positive on the exact figure) but it's probably less expensive than their current training runs. This is one of relevant papers.

We store tensors such as weights and the KV cache in on-device high-bandwidth memory (HBM). While there are other tensors that pass through the HBM, their memory footprint is much smaller, so we focus on just these two largest groups of tensors. ... Even though the computational cost of attention is relatively small, it can still account for a sig- nificant fraction of memory capacity and bandwidth costs, since (unlike the weights) the KV cache is unique for each sequence in the batch. ... For a 500B+ model with multihead attention, the attention KV cache grows large: for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model’s parameters.

Also, contra @xanados I'd say they probably don't run those models in bf16, so make that 180 GB.

@Porean investigated a similar question recently.

Re: your question – I also don't know of any convenient one-click-install local LLM-based applications (at least for natural language-focused models). There are not-terrible models you can run locally (nothing close to ChatGPT, to be clear) but you'll have to fiddle in CLI for now. There is no rush to develop such apps, because, again, models of practical size won't fit on any consumer hardware. Text is vastly less redundant than images.

I hope people realize soon the risk of offloading their brain extensions to Microsoft cloud and learn to pool resources to economically rent servers for community-owned LLMs, or something along these lines.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats