This thread is for anyone working on personal projects to share their progress, and hold themselves somewhat accountable to a group of peers.
Post your project, your progress from last week, and what you hope to accomplish this week.
If you want to be pinged with a reminder asking about your project, let me know, and I'll harass you each week until you cancel the service.
Jump in the discussion.
No email address required.
Notes -
Last week I whined that I really should have gone the Full Monty right away on my graphics card, mostly because I'm having so much fun dicking around with local LLMs but want more out of them. Today I stumbled upon the elegant solution of buying a used 3090 and a 1200 watt power supply (which I would have needed for the better card anyway) to get myself to a nice respectable 40 gigs of VRAM, which is oh-so-tantalizingly close to being able to run a full 70b model at the desired Q4_K_M. Odin dammit but do I ever wanna pull the trigger and spend another ~$1500 to net myself that headroom!
That is all. For now...
What's the best I can do with a 32gb gpu? I've been running gemma 4 31b in LM Studio which lets me load the model into vram. Loose rumors say you can fit more into it, or use ram (64 gb) or the cpu.
Gemma4 is quantization sensitive enough that 31B-Q5 is probably the best bet, closely followed by Gemma4 26BA4B around Q4-Q6 where you want (much) higher speed and simpler questions. That said, Gemma4 is great for writing, mediocre for coding, and lackluster for complex spatial reference.
Qwen 3.x is really good for local coding, especially 3.6 35B-A3B and 3.6 27B. Depending on context length, you'll want either Q6 or Q5, make sure to get the MTP variant setup where available. If you just want to code, these two models honestly cover 99.9% of anything you'd be able to do locally with any practical single-GPU machine.
Nemotron 3 Nano Omni 30B-A3B (Q4-Q5) is a little weird and it's not as good for code generation as Qwen, but it does handle really big contexts for analysis better. It's a bit annoying to set up properly, though.
Llama.cpp can run GGUFs of models in mixed-inference mode, where some layers operate on the GPU and some run in system RAM. This can be slightly slower for MoE models (anything with AXB as a suffix), or much slower for dense models, but it lets you run stuff that'd otherwise be impossible. I've got GLM-4.5-Air running on a single nVidia 3090 and 128 GB system RAM at Q8... admittedly, at <5 tokens/sec. Still can be useful for things like draft review if you let it run overnight.
((If you're really desperate, you can even offload to NVME, but this is a very bad idea for drive wear reasons.))
The general rule-of-thumb is that even if you're willing to accept slow inference, quants under Q2 are usually useless and under Q3 are marginal at best, so this doesn't mean it's worth the bandwidth and drive space to go with something like GLM-5.2 at Q1 just because the parameter count is high. But there are some useful options, still. For your setup, some that might be worth evaluating:
More options
Context Copy link
With the caveat that I am totally not an expert in this situation, it sounds like you're right in the sweet spot, as 31b will fit comfortably into 32 gigs at Q5 with a little bit of headroom. If it were me, I'd probably be sticking with 32b myself, so I'd be right there with you.
Right, though I've heard that with quantization - reducing the demand for accuracy in the memory's calculations, you can cut a model size effectively in half or more and fit something much bigger into vram.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link