Local Open-Weight Language Models on Consumer Computers
Hardware Tiers, Measured Trade-offs, and the Frontier-Cloud Boundary
tl;dr: pick the smallest model that passes your task, keep it local, escalate on purpose.Abstract
Local inference should be judged on four things: whether a model fits with memory to spare, how fast it runs, whether it is capable enough for the task, and how it compares with hosted frontier systems. This paper proposes a hardware tiered framework for PCs and Apple silicon Macs, built from official documentation, recent systems research and an original benchmark on a 32 GB M1 Pro. There is no universal winner against OpenAI, Anthropic or Google services. A local first design with explicit cloud escalation is often the most defensible choice.
Generation speed on a 32 GB M1 Pro
Qwen3.5 · Q4_K_M · llama.cpp · 128 tokens
Three hardware tiers
Low · 4 to 12 GBSmall models for extraction, classification and short summaries.
Mid · 12 to 32 GBThe strongest general local option: private RAG, coding help, adapters.
High · 32 to 128+ GBBigger models become possible, but capacity is not frontier quality.
What's inside
- 01What must fit. Weights, KV cache, runtime buffers and headroom, and why a 9B model needs more than the napkin math says.
- 02The M1 Pro case study. Prompt processing and generation for three model sizes, five repetitions each.
- 03Local versus frontier. Capability, reliability, cost structure, context and concurrency.
- 04A decision framework. Recommendations by tier and a minimum acceptance test before you ship.