I've got 128gb of DDR4 ram and an HP Z840 workstation. I want to get started on experimenting with local LLM just for curiosity, what GPUs do you recommend? I am eyeing Nvidia quadro p5000s, I can get two in the workstation easily for 32gb of vram. Thinking of running MoE models so the active is in vram. Also thinking of strata engine + Qwen models.
Anyone have other recommendations regarding GPU and model choice? I'm not looking to spend much money on this, it's a side project. Say under $1000 budget for add-ons to my current setup.
The goal is to rival frontier models on my narrow domain of physical science/applied math, and exceed frontier models in life administration/management by providing the model highly personal diaries, journals, and information I have found over the years in niche personal topics that are probably not well-sampled by frontier training data.
Obviously not going to rival the internal ones. But if we assume the AI engineers are doing something special with their internal models, that makes them a step beyond the frontier we access for $20/month or $200/year, then can I do those same things to my local models to take them from a step behind the $20/month frontier to being equal.
Anyone else having success with local models and want to share their setup, and also, your workflow? I think the workflow and meta-architecture is our edge over the frontier.
p
I've got 128gb of DDR4 ram and an HP Z840 workstation. I want to get started on experimenting with local LLM just for curiosity, what GPUs do you recommend? I am eyeing Nvidia quadro p5000s, I can get two in the workstation easily for 32gb of vram. Thinking of running MoE models so the active is in vram. Also thinking of strata engine + Qwen models.
Anyone have other recommendations regarding GPU and model choice? I'm not looking to spend much money on this, it's a side project. Say under $1000 budget for add-ons to my current setup.
The goal is to rival frontier models on my narrow domain of physical science/applied math, and exceed frontier models in life administration/management by providing the model highly personal diaries, journals, and information I have found over the years in niche personal topics that are probably not well-sampled by frontier training data.
Obviously not going to rival the internal ones. But if we assume the AI engineers are doing something special with their internal models, that makes them a step beyond the frontier we access for $20/month or $200/year, then can I do those same things to my local models to take them from a step behind the $20/month frontier to being equal.
Anyone else having success with local models and want to share their setup, and also, your workflow? I think the workflow and meta-architecture is our edge over the frontier.
More options
Context Copy link