Local AI Computers in 2026: Real LLM Benchmarks, Not NPU TOPS

Compact computers can now run surprisingly large language models locally. But fitting a model into memory is not the same as running it quickly, and advertised NPU TOPS are a particularly poor substitute for measured LLM throughput.

This comparison focuses on single-user local inference and coding agents, not training, synthetic TOPS, or multi-user datacenter throughput. The figures below are published third-party measurements, not tests I performed. Because hardware, models, quantization, context, backends, and configurations differ, a row from one test is not necessarily directly comparable with a row from another.

Hardware at a glance

The price column is deliberately relative, not a quotation: $ is the least expensive class; $$$$$ is the most expensive. It reflects broad complete-system purchase tiers, not exact live prices in a particular country. Memory figures describe installed or possible system memory, not necessarily memory that an inference backend can allocate to the GPU.

Caution: Price symbols are coarse editorial categories. The actual cost of an HX 370 barebone changes substantially after adding 64–96 GB RAM and storage. Machines sharing a chip are grouped for architectural comparison, not credited with identical measured results.

Published generation benchmarks

tg128 generally means generation of 128 output tokens in llama-bench; pp512 is the speed of processing a 512-token prompt. Both are in tokens per second. Some sources used server-based decoding rather than tg128, as explicitly marked.

N/A means no sufficiently documented measurement was selected, not that the machine cannot run the model. No numerical benchmark is assigned to an untested commercial chassis merely because it has the same SoC. The DGX Spark source should be consulted for the exact model files and quantizations before making a strict same-model comparison.

The one model that appears across most of these sources is Qwen3 30B-A3B, so it is the closest thing to a common yardstick:

What these measurements actually tell us

1. MoE versus dense is the decisive split

On the same Strix Halo test systems, Qwen3 30B-A3B runs at 86.1 tok/s, while Llama 3.1 70B dense runs at 5.1 tok/s. A 120B MoE can also be faster than a much smaller dense model. Total parameter count alone is a misleading performance metric: active experts, memory traffic, quantization and implementation matter.

The same pattern holds on every machine with both kinds of result:

2. The large-memory Strix Point boxes are capable, but not equivalent to Strix Halo

Sapphire EDGE AI 370, Minisforum AI X1 and GEEKOM A9 Max use Radeon 890M-class graphics, whereas Strix Halo machines use the much larger Radeon 8060S and higher-bandwidth memory. An HX 370 test measured 28–31 tok/s on a 35B-class quantized MoE, but this result came from a different HX 370 device with LPDDR5X; a Sapphire system with DDR5 SO-DIMMs may behave differently. It would be misleading to call this a measured Sapphire result.

3. The RTX 5090 is extremely fast when the model fits

The measured 365.73 tok/s for Qwen3 30B-A3B Q4_K_M on an RTX 5090 is impressive. But the card’s 32 GB VRAM constrains model choice, context length and concurrency. A slower 128 GB unified-memory box can therefore be more useful for a substantially larger model, even though it loses a same-model speed contest.

4. Apple Silicon deserves a separate look

A published M5 Max llama.cpp test reaches 148.5 tok/s on Qwen3 30B-A3B Q4_K_M, with the measured 48 GB machine also reaching 26.4 tok/s on Qwen3 32B dense. That is a strong result for a compact single-user machine. Do not automatically transfer these numbers to M4 or to every M5 Max memory/GPU configuration.

5. Coding agents care about prefill, latency and context, not just short decoding

A particularly useful long-context study on Beelink GTR9 Pro shows Qwen3 30B-class MoE decoding changing from 88.4 tok/s on an 11-token prompt to 30.1 tok/s on a 48K-token prompt. Processing that 48K-token input took approximately 189 seconds in that particular setup. An attractive tg128 result can hide very poor agent-loop latency once a repository, tool schemas and conversation history are included.

Source: Strix Halo, measured. This is a particular backend/server/model configuration, not a universal prediction for every Strix Halo deployment.

How I would benchmark a coding-agent machine

Before choosing hardware, I would run at least two models: one small dense model that fits in GPU memory and one large MoE suited to code and tool use. Record:

1. pp512, pp4096 and pp16384 — prompt-processing throughput.
2. tg128 at short context — baseline single-request decoding speed.
3. End-to-end latency at 32K and 64K context — time to first token, decoded tok/s and total response time.
4. Peak allocated GPU/system memory — including KV cache, not just the model file.
5. Power consumption and sustained behavior — short peaks can obscure thermal throttling.

For a reproducible baseline with llama.cpp:

llama-bench -m /models/model.gguf -ngl 99 -p 512 -n 128 -r 3

The command is only a starting point. Document the model hash, quantization, llama.cpp commit, backend, GPU offload settings, flash attention, context, power limit and operating system. For an actual agent, measure the HTTP server and real prompt sizes too.

Bottom line

• Budget CUDA box: RTX 5060 Ti 16 GB. Suitable for smaller models that fit in VRAM; evaluate VRAM before headline throughput.
• Affordable high-memory experimentation: Sapphire EDGE AI 370 and other HX 370 mini-PCs. They are genuine local-inference machines, but performance claims need same-device validation.
• Large local MoE models: 128 GB Ryzen AI Max+ 395 machines such as EVO-X2 and GTR9 Pro, with published results for 80B and 120B-class MoE workloads.
• Compact high-throughput Apple option: Mac Studio M5 Max, with unusually strong published single-user results on supported model sizes.
• NVIDIA software stack with 128 GB unified memory: DGX Spark, whose primary value is not necessarily the best short-prompt decode speed per price tier.
• Maximum speed for models fitting in 32 GB VRAM: RTX 5090; dramatically stronger throughput in the cited Qwen3 MoE test, but a different memory-capacity tradeoff.

The single most important purchasing question is not “How many billions of parameters can this box load?” but “How long will my actual agent take to process 32–64K of context and complete a tool-use cycle?”

Sources and reproducibility

• Strix Halo LLM Performance — EVO-X2 and GTR9 Pro
• Detailed Strix Halo benchmark matrix
• Strix Halo long-context measurements
• Ryzen AI 9 HX 370 Vulkan/HIP measurements and platform-support discussion
• Mac mini M4: reproducible local LLM benchmark
• Mac Studio M5 Max: reproducible LLM benchmark dataset
• DGX Spark llama.cpp benchmark report
• RTX 5090 Qwen3 30B-A3B single-card comparison
• RTX 5060 Ti benchmark compilation
• llama.cpp

Last reviewed: October 16, 2026. Hardware models and software backends evolve quickly; reported results are snapshots, not manufacturer guarantees.