SYSTEMS NOTE / 004
PROJECT 04 · EDGE AI · SEPTEMBER 2026
We compare four small models on consumer-level hardware.
01 / HARDWARE USED
No discrete GPU. Every model ran locally through llama.cpp.
02 / MODELS USED
8.3B total / 1.5B active; convolution and attention layers.
7B total; Mamba-2, attention, and routed experts.
8.2B dense Transformer rather than routed experts.
3B dense Transformer; the smallest model in the comparison.
GGUF packages each model for llama.cpp; Q4_K_M is the common, mostly 4-bit K-quantization—not another model architecture.
All four also used the same 4,096-token context.
01 / THE QUESTION
Beyond the model name, performance depends on quantization, runtime, context, hardware, prompts, tool schemas, and the validation around the response.
Our narrower question was: on this exact stack, which model fits which edge workload? Semantic success asks whether the decision is correct. Contract success asks whether software can safely consume it without interpretation.
02 / RESULTS
The benchmark is a comparison of complete configurations, not model names in the abstract. We fixed llama.cpp build b10964, Q4_K_M quantization, 4,096-token context, one parallel slot, temperature zero, seed 42, the harness, and the Ryzen 3 5300U. Only the loaded model changed.
More consistent exact outputs; dense architecture increased CPU decode cost.
Fast CPU decode; exact outputs often required normalization.
Moderate memory use with comparatively consistent tool routing.
Lower memory use; structured tool routing failed in this configuration.
READ THIS CAREFULLY “Semantic” gives credit for the correct decision inside extra prose. “Contract” requires the exact requested output or a real structured tool call. Tool-call-shaped text receives no credit.
Qwen3-8B is dense: its blocks do not route around inactive experts. On this memory-bandwidth-limited CPU, more model data had to move for every generated token. See the scorecard for the measured decode rate.
LFM2.5-8B-A1B is a hybrid MoE that activates only a subset of its parameters per token and uses convolution-heavy layers. Less active work plausibly explains the decode-rate difference in the scorecard; this run did not isolate kernel effects separately.
03 / USE-CASE RESULTS
Why the Qwen result moves sharply It passed one of four edge-math tasks—the timestamp update—and missed temperature conversion, compound energy growth, and anomaly thresholding. With only four tasks, each result moves the score by 25 points; this is a failure pattern, not a broad claim about math ability.
Each bucket represents a failure surface in an edge system—not a school-subject score.
04 / OUTPUT COMPLIANCE
Each gap subtracts exact contract success from semantic success. It shows how often an answer was substantively correct but not usable by the required machine interface because of formatting or tool-call compliance. The scorecard contains the underlying rates; a smaller gap implies less adapter and validation work.
This is a system-design choice, not a model leaderboard: either pair a fast, semantically capable model with validation and a narrow adapter, or accept lower speed for a model whose native interface behavior better fits the control loop.
05 / MODEL COMPARISON
06 / METHOD AND LIMITS
28 deterministic edge tasks
Sensor decisions, local math and extraction, offline grounding, compression, tool routing, and privacy controls.
112 task evaluations
The same suite was executed once against each of four locally loaded Q4_K_M models.
Two scoring layers
Semantic correctness is separated from exact machine-contract compliance. Tools count only when emitted as executable structured calls.
One pinned stack
AMD Ryzen 3 5300U, llama.cpp build b10964, Q4_K_M weights, 4K context, one slot, temperature 0, seed 42, and the same prompts and tools.
Comparable throughput run
Three uncached 3,800-token prefills plus 128-token decodes per model. Inference requests were rejected unless addressed to localhost.
These results answer one bounded question: how these four Q4_K_M models behaved inside this harness on this machine. They do not predict another runtime or device. The next experiment should change one layer at a time—seed, quantization, context, or hardware—and add private workloads from the intended deployment.