The Model Was Right. The Backend Wasn’t.

A Qwen3.6-27B instance running a network automation workflow last week configured GigabitEthernet0/1/4 instead of GigabitEthernet0/0/1.201. The weights were correct — a BF16 reference run on the same model completed the task without error. What changed was the attention backend. Under FlashAttention 2 at roughly 100,000 tokens of context, the model diverged from its reference output and executed the wrong interface configuration twice, in separate tool calls, with independent branching. The model card was fine. The CUDA kernel wasn’t.

This is the gap in how local LLMs get evaluated. Model quality discussions focus on weights: quantization level, parameter count, benchmark scores. A Q4_K_M file sits next to a Q8_0 file and the community debates perplexity loss. What doesn’t get measured is the execution environment — which CUDA kernel, which attention backend, what KV cache precision, how tensor parallelism interacts with the model’s NCCL graph. These choices produce measurable divergence from reference outputs. In chat contexts, the divergence is invisible. In agentic contexts, it is functional failure.

The Part of the Stack Nobody Benchmarks

AWQ — Activation-aware Weight Quantization won the MLSys 2024 Best Paper Award. The MIT HAN Lab paper benchmarks AWQ on language modeling, coding tasks, and math — the standard suite. It achieves 2.7x speedup over FP16 on an RTX 4090 with, the paper claims, minimal accuracy degradation. NVFP4, Nvidia’s mixed-precision format for Blackwell GPUs, promises a similar tradeoff: significant efficiency gains at the cost of reduced precision. Neither paper includes a test of tool-calling reliability, because tool-calling is not part of the standard evaluation suite. The benchmarks measure what they measure; the consequences downstream are someone else’s problem.

A researcher on the Level1Techs forum spent three weeks finding out what the stack actually does to a running agent. The methodology is serious: 44.8 terabytes of stored logit data, complete logit capture during every tool call execution, teacher-forced decoding that traces exactly what happens from the point a token diverges. The model is Qwen3.6-27B on an RTX PRO 6000 Blackwell. Five quantization schemes, three attention backends, four KV cache precision levels, and four uncensored fine-tuned derivatives — all measured against a network automation workflow executing real Cisco commands. The test is specific, not abstract: a task that either configures the right interface or configures the wrong one.

The reason benchmark evaluations don’t catch these failures is structural. Standard evals run on reference hardware with reference implementations — the same inference framework the lab used when it generated the benchmark numbers. When a researcher downloads a GGUF and runs it through Ollama or llama.cpp with a different attention backend and a quantized KV cache, they are running a different execution graph on the same weights. The benchmark number doesn’t transfer. It was measured somewhere else, by a different stack, on a different hardware configuration that nobody documented on the model card. FlashAttention-3, for instance, is currently Hopper-specific and ships in beta; the AMD ROCm ecosystem already has two co-existing backends — Composable Kernel and Triton — that produce different outputs. Each hardware generation adds forks the standard benchmark never touched.

What the Stack Does to the Model

The quantization results from the Level1Techs analysis split into two tiers. BF16 reference, FP8 (E4M3), and INT8 W8A16 all complete tool calls correctly. FP8 flips roughly 40% of top-1 token selections by 88,000 tokens of context compared to BF16; INT8 flips roughly 20%. Both pass. NVFP4 and AWQ W4A16 fail. NVFP4 flips approximately 50% of tokens; AWQ flips approximately 45%. Both execute show run instead of show arp — the same wrong Cisco command — because they diverged from the reference output at a decision point deep in the tool call. The model was asked to look up a MAC address table. It ran a configuration dump instead.

The counter-intuitive result is that flip rate doesn’t predict functional failure in a linear way. FP8 at 40% flips still passes every tool call. AWQ at 45% fails every one. The difference is where in the probability distribution the flips land. The failing schemes flip tokens at positions where the stock model has high confidence — decisions the model has essentially locked in — while the passing schemes diverge at lower-confidence positions where alternative tokens are semantically close. A flip from one preposition to another in a sentence summary is noise. A flip from show arp to show run when the model has 99.9% confidence in its choice is a functional inversion. The flip rate measures quantity; what matters is which decisions get overturned.

KV cache precision has a separate effect with a sharper cliff. INT4 KV cache causes complete tool call failure — the model cannot terminate function envelopes correctly. INT8 eventually recovers after divergence. BF16 runs clean. Tensor parallelism adds a third variable: TP1 executes correctly, TP2 fails tool calls, TP4 succeeds again — a consequence of how NCCL graph construction differs across parallelism configurations that use the same weights. None of these are exotic settings. They are the standard configuration options exposed in vLLM and llama.cpp. The defaults aren’t always BF16; the documentation describes options, not consequences.

When Configuration Becomes Consequential

For conversational workloads — summarize this document, explain that concept — the divergence is invisible. A model with 45% token flips relative to its BF16 reference still produces a coherent response. The specific words change; the meaning doesn’t break. This is why the community discussion about quantization has stayed comfortable for years. Users have been running quantized models for chat, and chat tolerates imprecision gracefully. Nobody noticed the Cisco command was wrong because nobody was running Cisco commands.

Agentic workloads don’t tolerate imprecision. Anything that executes commands, constructs structured JSON, calls functions, or writes to external systems fails hard rather than gracefully — it terminates the wrong function, sends the wrong parameter, leaves the tool envelope open or malformed. This is the shift that makes the inference stack problem urgent now rather than theoretical. Local model deployment is accelerating for exactly the workloads that require precision: coding agents running bash, network automation tools configuring infrastructure, document processors writing records to databases. The hosted model handles reproducibility as a feature of the service — one stack, one implementation, consistent outputs across sessions. The local deployment doesn’t. The model card can’t fix this because it was benchmarked on a reference implementation that nobody running local agents actually uses.

The fine-tuned model data adds one more variable. The Level1Techs researcher tested four uncensored derivatives of Qwen3.8 — popular downloads in local deployment communities that remove content restrictions via abliteration or similar techniques. The most careful fine-tune, Heretic-ARA, flipped only 0.717% of tokens compared to the stock model — essentially faithful. The most aggressive, AEON Ultimate, flipped between 2.997% and 5.831% depending on system prompt context, produced between 8 and 36 invalid tool-call branches in testing, and corrupted a PostgreSQL port number to 543ql instead of 5432. The stock model had 0.9991 confidence in the correct digit; AEON selected an alternative with 0.9158 confidence. The model wasn’t uncertain. It was wrong with conviction, at a token where the stock model had near-certainty, because the fine-tuning shifted the probability mass in exactly the wrong direction for this task. The GGUF page on Hugging Face describes the capability improvements. It doesn’t document the logit drift.

The Accountability Gap

The FlashAttention repository documents that outputs match reference implementations “up to some numerical tolerance.” This is accurate. It doesn’t capture what “some tolerance” means at 100,000 tokens in an agentic context: wrong network interface, repeated twice in branching tool calls. The AWQ paper benchmarks on language modeling, coding (meaning code completion metrics, not structured output reliability), and math. The NVFP4 documentation describes efficiency gains on Blackwell hardware. None of these — the backend implementations, the quantization papers, the model cards — say anything about which configuration to trust for tool-calling in a 100k-token agentic loop. That information doesn’t exist in a single document, because nobody has been responsible for producing it.

The gap is structural: the people who build inference backends are optimizing for throughput and memory efficiency, not downstream task reliability across all possible workloads. The people who write model cards are benchmarking on the lab’s reference stack, which is not the stack anyone else uses. The people who publish quantization papers are targeting the metrics their reviewers will evaluate. Each group is doing something reasonable within their domain. The consequence falls on the developer who downloads a popular GGUF, sets KV cache to INT8 because the README says it saves memory, picks an attention backend because it benchmarks faster, and discovers three weeks later that their agent has been configuring the wrong database port in production.

The industry is building toward local agents: coding assistants that ship pull requests, network tools that configure infrastructure, document processors that write records to databases. Until inference stack benchmarking becomes a standard part of model evaluation — tool-calling reliability across backends and quantization schemes, KV cache precision at production context lengths, fine-tune logit stability against a reference baseline — local agentic deployment is an undocumented experiment. The model card lists what quantization level to download. The quantization level is not the right variable to watch.

AI-generated editorial illustration · TemperatureZero · August 23, 2026