There’s a lot of outdated advice floating around about what you need to run a local LLM well, so here’s where things actually stand for anyone weighing it right now.
For a genuinely useful coding/agent assistant running locally (not just a chat toy), you want enough VRAM to hold a mid-size model without heavy quantization loss. 24GB gets you comfortably into 30-40B territory at decent quality; 48GB+ opens up the larger open weight models people are actually using for real work.
CPU-only inference has gotten much better than it used to be, but it’s still a real latency tradeoff for anything agentic where you’re making many sequential calls — fine for a single Q&A, painful for a 20-step tool-use loop.
The unglamorous bottleneck people underestimate is disk I/O and RAM for model loading/swapping if you’re running more than one model, and power/cooling if you’re running this on a desk instead of a rack.
What’s your current setup, and what’s actually been the limiting factor for you: VRAM, throughput, or something else entirely?