I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)
I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM:
ASUS ROG MAXIMUS XI HERO Motherboard
Intel® Core™ i9-9900K × 16
64GB DDR5 RAM
RTX 3090 with 24GB VRAM
A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think
I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings.
I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.
I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.
The person you are responding to is not rich by any means in terms of tech standards. If you work at a big tech company (not even a FAANG), you make very decent money in good times and bad.
They could have financed this purchase with their yearly bonus or selling some equity that vested.
Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.
I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.
I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)
Qwen Next 3.8 @ 3 bit IQ on x2 Quadro 6000 (sm_75) on a Dell T5820 (W-2140B Xeon, 128GiB DDR3). Strata makes the most of the hardware.
heavily customised agentic framework: https://github.com/connollydavid/host gets good work out of the weights
I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM: ASUS ROG MAXIMUS XI HERO Motherboard Intel® Core™ i9-9900K × 16 64GB DDR5 RAM RTX 3090 with 24GB VRAM
A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think
I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings. I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.
I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.
How did you get so rich? Inheritance? Startup went IPO?
The person you are responding to is not rich by any means in terms of tech standards. If you work at a big tech company (not even a FAANG), you make very decent money in good times and bad.
They could have financed this purchase with their yearly bonus or selling some equity that vested.
Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
r9700s - running qwen 3.8 flash next with ~1m token kv cache
I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.
I got pretty good results with my 3090 on Qwen 27b. Yeah, the rack is completely unusable when you launch the llm, but i wouldn't say its slow.
Btw, fellow 3090 user, some time ago i saw this: https://github.com/Zagitta/gpu-unlock
Can't try it personally because windows, but maybe it will help you.
I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.
Upgraded from a 3060 12G to a 4070 12G.
I run a cluster of two Dell Pro Max GB10's 128GB unified mem per node 200G NIC ConnectX
what model can you run with those?