Local Open-Weight Language Models on Consumer Computers

Abstract

Local language-model inference is no longer a binary question of whether a model launches. The useful questions are which model fits with working memory left over, how quickly it processes prompts and generates text, whether its capability is sufficient for the task, and what is lost or gained relative to a hosted frontier system. This paper develops a hardware-tiered framework for answering those questions on consumer PCs and Apple-silicon Macs.

The synthesis combines official model cards, runtime documentation, current vendor specifications, recent systems research, and an original three-size benchmark on a 32 GB M1 Pro MacBook Pro. Under one reproducible llama.cpp configuration, Qwen3.5 Q4_K_M models at 2B, 9B, and 27B produced 62.8, 18.6, and 5.38 generated tokens per second, respectively. The result demonstrates a central distinction: a model that fits in memory may still be a poor interactive fit. The comparison with current OpenAI, Anthropic, and Google services finds no universal winner. Low-end local systems are best treated as private, offline appliances; mid-tier systems provide the strongest general-purpose local value; high-memory machines make larger models practical but do not automatically reproduce frontier-service quality, context handling, tools, or operational reliability. For many users, a local-first system with explicit cloud escalation is the most defensible architecture.

Keywords: local LLM inference, open-weight models, consumer GPU, Apple silicon, quantization, unified memory, llama.cpp, frontier models, privacy, inference economics

Introduction

Open-weight language models now span sub-billion-parameter edge models, dense models in the tens of billions, and sparse mixture-of-experts systems whose total parameter counts exceed what most consumer machines can hold. The surrounding software has matured with them. llama.cpp supports Metal, CUDA, HIP, Vulkan, SYCL, CPU execution, hybrid CPU/GPU offload, and quantization from 1.5 to 8 bits [1]. Qwen, Gemma, and gpt-oss publish downloadable weights across materially different size and architecture classes [3–7]. The result is a real local-computing option, not merely a research demonstration.

It is also easy to describe this option badly. Parameter count alone does not determine memory. File size alone does not determine usable context. Advertised context length does not mean that a consumer machine can serve that context. A Mac with unified memory and a PC with the same nominal number of gigabytes do not expose those gigabytes to the accelerator in the same way. A tokens-per-second number from one runtime, prompt length, quantization, and batch setting does not transfer cleanly to another. Finally, vendor benchmark tables are not controlled cross-vendor comparisons.

This paper therefore makes three contributions. First, it defines low, mid, and high local-inference tiers by usable accelerator-accessible memory rather than by retail label. Second, it measures the throughput curve of three current model sizes on one documented Mac configuration. Third, it compares local deployment with current hosted frontier systems across capability ceiling, privacy boundary, context, latency, cost structure, customization, and operational burden. The goal is a decision framework, not a leaderboard.

Scope and Method

Open weight is the accurate term

This paper uses open-weight for models whose trained weights can be downloaded and run under a published license. That does not imply that the training data, complete training code, or development process is open. Qwen3.5 uses Apache 2.0 [3–5], as does gpt-oss [7]. Gemma permits responsible commercial use under its own terms [6]. License review remains part of deployment engineering.

Evidence hierarchy

Hardware and model specifications come from first-party documentation. Runtime behavior comes from official project documentation and the local benchmark. Broader performance claims use recent systems papers, several of which are preprints and are identified as such [11–13]. Vendor-reported capability scores are treated as claims about a vendor's own evaluation setup, not as proof of superiority over models tested elsewhere.

Tier definition

The tiers refer to usable accelerator-accessible memory: VRAM that a discrete GPU can use directly, or the practical GPU working set available from unified memory. Total system RAM is not interchangeable with this quantity. Apple exposes a recommended maximum GPU working set because the operating system and other applications still require headroom [9]. On discrete-GPU PCs, system RAM can support partial offload, but moving weights across the CPU/GPU boundary commonly sacrifices throughput [1, 13].

What Must Fit Before a Model Can Work

A useful first approximation for weights is parameter count multiplied by bits per weight, divided by eight. A nominal 9B model at four bits therefore begins near 4.5 GB. Real files and loaded models are larger because quantization formats store scales and metadata, some tensors remain at higher precision, and multimodal encoders may add parameters. The 9B Q4_K_M file in this paper measured 5.67 GB, not 4.5 GB.

Quantization changes capacity and sometimes quality

Post-training quantization reduces the storage and bandwidth required per weight. GPTQ demonstrated accurate one-shot weight quantization at three or four bits for large transformer models [14]. QLoRA showed that a frozen four-bit base can support parameter-efficient fine-tuning through low-rank adapters [15]. Neither result means that every four-bit format is interchangeable. Accuracy, kernel support, load time, and throughput depend on architecture, quantizer, group size, calibration data, and hardware. A recent Apple-silicon study explicitly cautions that fewer bits do not guarantee faster inference on every platform [12].

Context consumes working memory

Autoregressive inference stores keys and values from prior tokens so later tokens do not recompute the entire history. This KV cache grows with sequence length and concurrent sequences. Systems such as vLLM use paged memory management to reduce fragmentation and improve serving utilization [16]. Local desktop runtimes also support lower-precision caches, but a 128K or 262K context advertised by a model card is an architectural limit, not a promise that a specific laptop can use it efficiently. Long-context planning must specify the target prompt length, batch size, cache precision, and available headroom.

Three Consumer Hardware Tiers

The boundaries below are deliberately ranges rather than product names. A high-end gaming label can still hide a hard VRAM limit, while an older Mac with more unified memory may hold a larger model but generate more slowly. The right tier is the one that supports the target workload with headroom.

Low-end systems: useful appliances, not universal assistants

Current small models are more capable than parameter count once implied. Qwen3.5 publishes 2B and 9B checkpoints with multimodal input, tool-oriented post-training, and native 262K context architecture [3, 4]. Gemma 4 publishes E2B and E4B variants whose official Q4_0 memory estimates are 2.9 and 4.5 GB, including a stated 20 percent loading overhead [6]. These models make local extraction, classification, summarization, and constrained assistance plausible on ordinary machines.

The limit is not merely speed. Smaller models have less capacity for difficult reasoning, ambiguous instruction following, and long-horizon agents. A low-end machine is strongest when the workflow narrows the task, validates structured output, retrieves authoritative context, and permits escalation. It is weakest when users expect a single unconstrained model to match a current frontier service across coding, research, multimodal reasoning, and tool use.

Mid-tier systems: the practical local sweet spot

Between roughly 12 and 32 GB of usable accelerator memory, local inference becomes a general engineering option. Dense 7B–14B models can run at moderate quantization with room for context. Sparse models can provide more total capacity while activating fewer parameters per token, although all experts still need to be stored. OpenAI states that gpt-oss-20b, a 21B-parameter mixture-of-experts model with 3.6B active parameters per token, runs within 16 GB in its native MXFP4 format [7, 8]. Google estimates 14.4 GB for Gemma 4 26B A4B at Q4_0, while noting that all 26B parameters must remain loaded despite only 4B being active [6].

This tier is where local RAG, private coding assistance, reusable adapters, and structured tool loops become credible. It is also where model choice matters most. Loading the largest available checkpoint can turn an otherwise responsive workstation into a five-token-per-second system. A smaller model with a better task fit, cleaner post-training, and higher usable precision often produces a better product.

High-end systems: capacity rises faster than certainty

A 32 GB RTX 5090 is a high-end consumer GPU [10], but 32 GB is still a hard boundary for fully resident weights and cache. OpenAI's gpt-oss-120b requires approximately 80 GB in native MXFP4 [7, 8]. High-memory Apple systems take a different path: the CPU and GPU share one memory pool [9], allowing larger models to remain GPU-accessible without crossing PCIe. Recent systems work reports that this capacity advantage can make very large quantized models practical on Apple silicon, while NVIDIA hardware retains stronger compute density and mature high-throughput serving paths [12, 13].

High tier does not mean frontier equivalence. Larger local models can improve reasoning and domain coverage, but their training, post-training, modalities, tool integration, safety layers, and update cadence remain different. They also move operational responsibility to the owner: model provenance, prompt templates, quantization validation, sandboxing, observability, updates, and incident response.

Case Study: Three Model Sizes on a 32 GB M1 Pro

Experimental setup

The test machine was a 2021 MacBook Pro with a 10-core Apple M1 Pro, 16-core GPU, and 32 GB unified memory. The runtime was llama.cpp release b10361, commit 14e78ddef, using its Metal and Accelerate backends with automatic full GPU offload. The checkpoints were Qwen3.5 2B, 9B, and 27B in Unsloth Q4_K_M GGUF files [17–19]. Their measured files were 1.27, 5.67, and 16.73 GB.

Each model ran five repetitions of llama-bench prompt processing at 512 tokens and text generation at 128 tokens. The benchmark reported the arithmetic mean and standard deviation. It excludes tokenization and sampling time [2]. No claims about answer quality follow from this throughput test; the purpose is to isolate the model-size curve on one machine and runtime.

Interpretation

Generation throughput fell by 70 percent from 2B to 9B and by a further 71 percent from 9B to 27B. Prompt processing fell even more sharply. The 27B model loaded and completed every test, but it generated roughly one token every 186 milliseconds. That can be acceptable for a careful single-user assistant. It becomes expensive in wall-clock time when an agent makes many serial calls, regenerates long plans, or performs tool loops.

The 9B checkpoint is the practical balance on this machine: it retained comfortable headroom and produced about 18.6 tokens per second. That is a hardware-specific conclusion, not a claim that 9B is the best capability point. A sparse model with fewer active parameters, an MLX-native conversion, speculative decoding, a different quantization, or a newer Apple GPU could shift the curve. Published Apple-silicon studies likewise find that runtime and quantization choices materially affect time-to-first-token and sustained throughput [11, 12].

Local Open Weights Versus Hosted Frontier Systems

A frontier API is not merely a larger checkpoint behind an HTTP endpoint. It is a managed system that can include proprietary post-training, request routing, reasoning controls, prompt caching, integrated search, file tools, computer use, multimodal encoders, safety classifiers, monitoring, and rapidly changing serving infrastructure. Conversely, a local model is not merely a cheaper API. It creates a different trust boundary and grants direct control over weights, runtime, retention, and adaptation.

Current hosted reference points

Table 3 is a dated specification snapshot, not a quality ranking. The providers use different architectures, reasoning accounting, tool charges, service tiers, and benchmark methods. The illustrative request cost uses 100,000 input tokens and 20,000 output tokens at standard listed token prices, without caching, batch discounts, grounding, or other tools.

Capability and reliability

Hosted frontier systems provide the safest default when failure is expensive, tasks are open-ended, or the workflow depends on strong tool use and long-horizon planning. Their providers can serve compute and memory far beyond one consumer machine and update the full system without the user rebuilding a runtime. The cost is dependence on a remote service, policy surface, network, provider lifecycle, and recurring metered use.

Local models are strongest when the task distribution is narrow enough to evaluate. They can be pinned to a version, adapted, instrumented, and run repeatedly without a per-token vendor fee. They also allow the operator to keep raw context on-device, provided the local application, tools, telemetry, and retrieval sources do not transmit it elsewhere. Locality is a system property, not a property of the weight file.

Cost structure

API cost is visible and elastic: use more tokens and pay more. Local cost is front-loaded and partly hidden: hardware depreciation, electricity, storage, engineering time, model evaluation, updates, and failures. A local break-even equation is therefore workload-specific:

local monthly cost = amortized hardware + electricity + operator time + expected failure cost.

Comparing that total with API spend requires measured prompt and completion volumes, not a claim that local tokens are free. Low-volume users often benefit from the API because they avoid capital cost. High-volume, stable, privacy-sensitive workloads can favor local execution once the task succeeds on a model that runs fast enough.

Context and concurrency

Frontier APIs currently advertise context windows around one million tokens for the representative systems above [20–22]. A local model may advertise a similar architectural limit, but the desktop must allocate the cache and wait for prompt processing. Concurrency compounds both. A single-user chat benchmark should never be reused as a multi-user capacity claim. The local advantage is deterministic ownership of capacity; the frontier advantage is provider-managed scaling.

A Decision Framework

Recommendations by tier

Low tier: start with a 2B–4B instruction model at four or five bits. Constrain the task, validate outputs, and keep a frontier fallback for difficult reasoning. Do not buy hardware solely because a 20B checkpoint has a stated 16 GB minimum; minimum loading memory is not a comfortable workstation budget.

Mid tier: benchmark a 7B–14B dense model and one efficient sparse candidate. Prefer the smallest model that passes an application-specific evaluation at the required latency. This tier is the strongest default for private RAG, coding assistance, document work, and domain adapters.

High tier: choose the workload before the machine. A single high-end PC GPU favors speed on models that fit; a high-memory Mac favors capacity and energy-efficient single-user inference; multi-GPU PCs add capacity at the cost of power, topology, and operational complexity. Validate long context and concurrency separately from single-stream generation.

Hybrid: keep retrieval, redaction, classification, caching, and easy generations local. Escalate only approved context when the local model signals uncertainty, violates a schema, fails a verifier, or encounters a task class known to require frontier capability. The escalation rule should be deterministic enough to test and inspect.

Minimum acceptance test

A deployment decision should include at least: task success on a held-out set; hallucination or unsupported-claim rate; schema validity; prompt-processing time; generation throughput; time-to-first-token; peak memory at the target context; quality after quantization; tool-call validity; offline and network-failure behavior; and the exact model, quantization, runtime, prompt template, and date. Without those fields, “runs locally” is not an engineering result.

Limitations

The original benchmark covers one Apple machine, one runtime build, one model family, one quantization family, one batch configuration, and short synthetic throughput tests. It does not measure answer quality, power, time-to-first-token including tokenization, long-context cache growth, multimodal throughput, concurrent serving, or PCs. It should be reproduced on the target machine before purchase or deployment.

The broader systems literature is moving quickly and includes preprints [11–13]. Vendor model cards and pricing pages can change after publication. Model-reported benchmark results may use different prompts, tool access, reasoning budgets, sampling, and contamination controls. This paper therefore avoids a universal capability rank and dates its hosted-model table.

The three hardware tiers are planning ranges. A model can cross them through more aggressive quantization, partial offload, speculative decoding, cache compression, or architecture-specific kernels. Those changes can alter quality and latency. Likewise, a Mac's total unified memory is not all available to the GPU, and aggregate VRAM across multiple discrete GPUs does not automatically behave like one flat memory pool.

Conclusion

Local open-weight inference is now practical across consumer hardware, but practicality has three levels. On low-end machines, small models are valuable private tools for bounded work. On mid-tier machines, local inference becomes a credible general-purpose subsystem and often the best balance of capability, speed, and control. On high-memory machines, larger models become possible, but capacity alone does not recreate the training, tools, context service, and operational envelope of a current frontier provider.

The M1 Pro results make the trade-off concrete. Moving from 2B to 27B increased the model file by roughly thirteen times and reduced generated-token throughput by roughly twelve times. The 27B model fit; the 9B model was the more comfortable interactive system. That distinction should guide local deployments: optimize for the smallest model that passes the task, not the largest model that boots.

The defensible default is local-first and evidence-driven. Keep data and routine work on the device when the model passes. Escalate difficult or high-consequence cases to a frontier service under an explicit policy. Re-measure when the model, quantization, context, runtime, or hardware changes.

References

1. ggml-org. (2026). llama.cpp: LLM inference in C/C++. GitHub repository.
2. ggml-org. (2026). llama-bench documentation. GitHub repository.
3. Qwen Team. (2026). Qwen3.5-2B model card. Hugging Face.
4. Qwen Team. (2026). Qwen3.5-9B model card. Hugging Face.
5. Qwen Team. (2026). Qwen3.5-27B model card. Hugging Face.
6. Google. (2026). Gemma 4 model overview and inference memory requirements. Google AI for Developers.
7. OpenAI. (2025). Introducing gpt-oss.
8. OpenAI. (2026). gpt-oss reference implementations. GitHub repository.
9. Apple. (2026). hasUnifiedMemory. Apple Developer Documentation.
10. NVIDIA. (2026). GeForce RTX 5090 specifications.
11. Rajesh, V., Jodhpurkar, O., Anbuselvan, P., et al. (2025). Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS. arXiv:2511.05502. Preprint.
12. Benazir, A., & Lin, F. X. (2025). Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective. arXiv:2508.08531. Preprint.
13. Javat, A., & Kazakov, A. (2026). Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference. arXiv:2605.00519. Preprint.
14. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. International Conference on Learning Representations.
15. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36.
16. Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles.
17. Unsloth. (2026). Qwen3.5-2B GGUF quantizations. Hugging Face.
18. Unsloth. (2026). Qwen3.5-9B GGUF quantizations. Hugging Face.
19. Unsloth. (2026). Qwen3.5-27B GGUF quantizations. Hugging Face.
20. OpenAI. (2026). OpenAI API models. Official OpenAI documentation.
21. Anthropic. (2026). Claude models overview. Claude Platform Documentation.
22. Google. (2026). Gemini 3.5 Flash model specification. Google AI for Developers.
23. Google. (2026). Gemini Developer API pricing. Google AI for Developers.

Source-status note: references 11–13 were preprints at the time of writing. Vendor specifications, prices, and model availability were checked on 11 August 2026 and may change. Benchmark artifacts in references 17–19 are third-party quantizations of the official Qwen weights.