This thread is for anyone working on personal projects to share their progress, and hold themselves somewhat accountable to a group of peers.
Post your project, your progress from last week, and what you hope to accomplish this week.
If you want to be pinged with a reminder asking about your project, let me know, and I'll harass you each week until you cancel the service.
Jump in the discussion.
No email address required.
Notes -
Last week I whined that I really should have gone the Full Monty right away on my graphics card, mostly because I'm having so much fun dicking around with local LLMs but want more out of them. Today I stumbled upon the elegant solution of buying a used 3090 and a 1200 watt power supply (which I would have needed for the better card anyway) to get myself to a nice respectable 40 gigs of VRAM, which is oh-so-tantalizingly close to being able to run a full 70b model at the desired Q4_K_M. Odin dammit but do I ever wanna pull the trigger and spend another ~$1500 to net myself that headroom!
That is all. For now...
What's the best I can do with a 32gb gpu? I've been running gemma 4 31b in LM Studio which lets me load the model into vram. Loose rumors say you can fit more into it, or use ram (64 gb) or the cpu.
With the caveat that I am totally not an expert in this situation, it sounds like you're right in the sweet spot, as 31b will fit comfortably into 32 gigs at Q5 with a little bit of headroom. If it were me, I'd probably be sticking with 32b myself, so I'd be right there with you.
Right, though I've heard that with quantization - reducing the demand for accuracy in the memory's calculations, you can cut a model size effectively in half or more and fit something much bigger into vram.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link