This thread is for anyone working on personal projects to share their progress, and hold themselves somewhat accountable to a group of peers.
Post your project, your progress from last week, and what you hope to accomplish this week.
If you want to be pinged with a reminder asking about your project, let me know, and I'll harass you each week until you cancel the service.
Jump in the discussion.
No email address required.
Notes -
Last week I whined that I really should have gone the Full Monty right away on my graphics card, mostly because I'm having so much fun dicking around with local LLMs but want more out of them. Today I stumbled upon the elegant solution of buying a used 3090 and a 1200 watt power supply (which I would have needed for the better card anyway) to get myself to a nice respectable 40 gigs of VRAM, which is oh-so-tantalizingly close to being able to run a full 70b model at the desired Q4_K_M. Odin dammit but do I ever wanna pull the trigger and spend another ~$1500 to net myself that headroom!
That is all. For now...
What's the best I can do with a 32gb gpu? I've been running gemma 4 31b in LM Studio which lets me load the model into vram. Loose rumors say you can fit more into it, or use ram (64 gb) or the cpu.
Gemma4 is quantization sensitive enough that 31B-Q5 is probably the best bet, closely followed by Gemma4 26BA4B around Q4-Q6 where you want (much) higher speed and simpler questions. That said, Gemma4 is great for writing, mediocre for coding, and lackluster for complex spatial reference.
Qwen 3.x is really good for local coding, especially 3.6 35B-A3B and 3.6 27B. Depending on context length, you'll want either Q6 or Q5, make sure to get the MTP variant setup where available. If you just want to code, these two models honestly cover 99.9% of anything you'd be able to do locally with any practical single-GPU machine.
Nemotron 3 Nano Omni 30B-A3B (Q4-Q5) is a little weird and it's not as good for code generation as Qwen, but it does handle really big contexts for analysis better. It's a bit annoying to set up properly, though.
Llama.cpp can run GGUFs of models in mixed-inference mode, where some layers operate on the GPU and some run in system RAM. This can be slightly slower for MoE models (anything with AXB as a suffix), or much slower for dense models, but it lets you run stuff that'd otherwise be impossible. I've got GLM-4.5-Air running on a single nVidia 3090 and 128 GB system RAM at Q8... admittedly, at <5 tokens/sec. Still can be useful for things like draft review if you let it run overnight.
((If you're really desperate, you can even offload to NVME, but this is a very bad idea for drive wear reasons.))
The general rule-of-thumb is that even if you're willing to accept slow inference, quants under Q2 are usually useless and under Q3 are marginal at best, so this doesn't mean it's worth the bandwidth and drive space to go with something like GLM-5.2 at Q1 just because the parameter count is high. But there are some useful options, still. For your setup, some that might be worth evaluating:
Thanks! I've looked up what these q levels mean. If you're doing coding, you'll want high precision and thus at least q5, whereas for everyday chat you could get away with q4 or even less?
I'm still unsure how model size (pre q.) plays into the final "iq" or usability, after pretty heavy q. has been applied, compared to a smaller model with lighter quantization.
It's a little weird, and it depends on what model you're using, what you're doing with it, and your set up. The general rule that if you have to go under Q4, you're probably better going off with a smaller model, but that's... very approximate.
Raw parameter count alone doesn't make for a smarter model, especially across families or generations. PALM was a 540B model, for example, and it'll make mistakes that modern models in the 100B range won't. With more recent models, raw parameter count matters less for reasoning ability, but smaller models will have far more and larger gaps in knowledge. As an example, Gemma4-12B has no idea what the Burned Furs controversy was and assumed it was about someone having heatstroke, Gemma4-26B-A4B talked about a real-but-different event from 2016, and GLM-4.5-Air knew the actual thing and had a summary equivalent to the actual furry wikis histories.
Mixture-of-expert models can be thought of as having a bunch of specialized sub-models and a coordination engine, so they're usually dumber than their headline number for general purposes, but smarter than their A#B parameter number would suggest for specialized purposes.
Quantization generally doesn't make the model dumb, so much as it makes it wrong. Hallucinations, weird word choices, different languages or character sets, or 'obvious' errors tend to pop up. I've heard Q2 described as a model being drunk, and it's not wrong. This varies a lot depending on model and use, though.
Be aware that there's the separate concept of KV cache quantization. That's a runtime memory toggle, not part of the model. Most models run KV_Q8 perfectly and KV_Q4 reasonably, but Gemma and some reasoning-heavy models are a little more 'correct' at KV_Q16 and goes outright wrong at KV_Q4.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link