Kimi K3 Closed the Coding Gap. The Hallucination Rate Is 51%.

On July 16, Moonshot AI published a blog post announcing the first open model to reach 3 trillion parameters. Kimi K3 runs 2.8 trillion total parameters with 50 billion active per inference pass, supports a 1-million-token context window, and — on Moonshot’s own benchmark table — leads Program Bench at 77.8 against GPT-5.6 Sol’s 77.6 and Claude Fable 5’s 76.8. On FrontierSWE, K3’s 81.2 outpaces GPT-5.6 Sol by nearly 10 points. Those numbers are real. The-Decoder’s analysis of Artificial Analysis data found that K3’s hallucination rate climbed from 39% on K2.6 to 51%. Claude Fable 5 sits at 16%. Both results come from the same architecture, and understanding the design decisions that produced them is the actual story.

A Model Engineered for Coding Efficiency

The central architectural innovation is Kimi Delta Attention — KDA — which Moonshot describes as a hybrid linear attention mechanism enabling 6.3x faster decoding in million-token contexts. The standard quadratic attention cost at a million tokens is prohibitive; KDA restructures the attention computation to scale subquadratically, which is what makes a 1M context window operationally real rather than a specification footnote.

Paired with KDA is Attention Residuals, or AttnRes, which Moonshot says selectively retrieves representations across depth rather than accumulating them uniformly. The stated result is a 25% improvement in training efficiency at under 2% additional compute. Both mechanisms address the same problem: how do you route information through a 2.8T parameter model without paying the inference cost of all 2.8T parameters on every token?

The answer is Stable LatentMoE. Kimi K3 activates 16 of 896 experts per token — a 1.78% activation ratio. For context: DeepSeek V3, the prior benchmark for open MoE efficiency, activates roughly 8 of 256 experts, a 3.1% ratio. K3 is more aggressively sparse. The routing mechanism, called Quantile Balancing, derives expert selection directly from router-score quantiles rather than heuristic load-balancing hyperparameters. This is cleaner than most MoE implementations, and likely more stable at 2.8T scale where hyperparameter sensitivity compounds.

The quantization approach is where the architecture gets unusual. K3 uses MXFP4 weights with MXFP8 activations — 4-bit microscaling for weights, 8-bit for activations — applied from the supervised fine-tuning stage onward. Most deployed quantization is post-training: you train at full precision and compress afterward. Moonshot baked the quantization into the post-training pipeline itself, which means the model was explicitly optimized to behave correctly under 4-bit weight precision during alignment. The payoff is operational: Moonshot reports greater than 90% cache hit rates in coding workloads, which is what the $0.30/MTok cache-hit input price reflects. The architecture is built around coding as a workload, not as a use case.

What the Benchmarks Actually Show

Moonshot’s benchmark table is more honest than most model release tables.

On Program Bench, K3 leads at 77.8, ahead of GPT-5.6 Sol (77.6) and Fable 5 (76.8). On Terminal Bench 2.1, K3 scores 88.3 — just short of GPT-5.6 Sol’s 88.8, and materially ahead of Fable 5 and Opus 4.8 at 84.6. On FrontierSWE, K3 scores 81.2, behind Fable 5’s 86.6 but ahead of GPT-5.6 Sol at 71.3 by a significant margin. On SWE Marathon (long-horizon software development) and OmniDocBench (vision-based document understanding), K3 leads both fields. The clearest deficit is DeepSWE, where GPT-5.6 Sol leads at 73.0, Fable 5 follows at 70.0, and K3 sits at 67.5.

Moonshot’s own blog acknowledges that performance “still trails Claude Fable 5 and GPT 5.6 Sol” overall. That’s technically accurate in aggregate, but it undersells what K3 actually achieved: on the coding evaluations most relevant to software engineering work, an open model is now within single-digit percentage points of both closed frontier models, and leads some of them outright.

Artificial Analysis runs a broader evaluation — nine benchmarks composited into an Intelligence Index. They currently rank K3 fourth out of 189 models, ahead of nearly everything except the closed frontier. Their notation of K3 as “somewhat expensive relative to peers” and flagging of high verbosity (130 million output tokens versus a 63M median for comparable models) are worth noting. The verbosity is consistent with the “excessive proactiveness” that Moonshot themselves list as a known limitation.

The 51% Problem

The AA-Omniscience Index measures factual knowledge reliability and penalizes hallucinations without crediting hedges — a score from -100 to 100, where a model gets credit for being right and loses ground for confidently stating a wrong answer. Kimi K2.6 scored a 6 on this index, with 33% accuracy and a 39% hallucination rate.

Kimi K3 improves the score to 18 and pushes accuracy to 46%. The hallucination rate rises to 51%. The-Decoder’s reporting of the Artificial Analysis data compares this to Claude Fable 5, which scores 40 on the same index — 61% accuracy and a 16% hallucination rate. Fable 5 is correct more often and wrong with confidence far less often. That gap is what separates a model that competes on coding benchmarks from a model that handles knowledge work reliably.

The architecture explains the direction of the tradeoff. MXFP4 at 4-bit precision compresses the weight space significantly; factual recall stored in lower-resolution representations degrades when queries land outside the training distribution that was most heavily represented. The 1.78% expert activation means that on any given token, 98.22% of K3’s parameters are dormant. The 16 activated experts reflect a narrow slice of the training data — one that, given the coding-first architecture and cache optimization, skews heavily toward code. When the question is about geography, history, or scientific fact rather than API syntax and algorithm design, the 16 activated experts are probably the wrong 16.

Developer Simon Willison ran his standard benchmark on K3 on release day: a prompt asking the model to generate an SVG of a pelican riding a bicycle. He published his findings the same day. The prompt consumed 13,241 reasoning tokens to generate 3,417 response tokens — a nearly 4:1 reasoning-to-response ratio — costing $0.25 for a task that runs for pennies on every other frontier model. Willison also identified what appears to be an undisclosed 85-token hidden system prompt, inflating per-call costs in ways that aren’t visible at the API level. The “excessive proactiveness” that Moonshot flags in their own release notes maps directly to this behavior: K3 reasons extensively, even when the task doesn’t warrant it.

The hallucination direction and the verbosity direction both follow from optimizing for long-horizon coding. Reasoning burn is a feature when you’re debugging a 10,000-line codebase across a million-token context. It’s a cost when you’re answering a factual question. The model doesn’t distinguish between them at inference time — it just reasons hard.

The Price That Used to Win

Kimi K3 costs $3.00 per million cache-miss input tokens and $15.00 per million output tokens. Claude Sonnet 5 — Anthropic’s current mid-tier model, described as “the best combination of speed and intelligence” — costs the same: $3.00 input, $15.00 output, with an introductory price of $2.00/$10.00 running through the end of August. Claude Fable 5, Anthropic’s most capable publicly available model, costs $10.00 and $50.00 per MTok. GPT-5.6 Sol is $5.00 and $30.00.

Kimi K2.6 cost $0.95 input and $4.00 output. K3 is a three-times price increase. The-Decoder’s analysis frames this correctly: Chinese AI providers are abandoning the race to the bottom. The price point that made DeepSeek and Kimi attractive to developers building cost-sensitive pipelines — the $0.27/MTok input cost for DeepSeek V3 a year ago, the sub-dollar Kimi K2 pricing — is not the positioning that Moonshot is chasing with K3. At $3/$15, K3 is priced to compete with the mid-tier Western models on capability, not to undercut them on price.

The Artificial Analysis data suggests K3 performs comparably to Opus 4.8, which is priced at $5.00 and $25.00. If that comparison holds, K3 is underpriced relative to its intelligence tier. The hallucination rate is probably why it’s not priced higher: you pay Fable 5 rates when you need factual reliability alongside capability. At 51% hallucination on AA-Omniscience versus Fable 5’s 16%, the premium isn’t justified for knowledge-work deployments.

The weights release on July 27 with the full technical report. The first thing the community will run is the hallucination rate at full BF16 precision. If the reliability regression is a precision artifact from MXFP4 inference, it should improve at full precision. If it’s a training-distribution artifact — the alignment phase ran under MXFP4 constraints on a code-heavy corpus, and the factual recall degraded regardless of inference precision — it won’t. Given that Moonshot applied MXFP4 quantization from the SFT stage onward, the second explanation is more likely. The model’s alignment phase happened under 4-bit constraints. That’s not a post-hoc approximation; it’s baked in.

Kimi K3 is the clearest evidence yet that the frontier gap is not about parameter count. You can build the largest open model in history — 2.8 trillion parameters, 50 billion active, the most aggressively sparse MoE ever deployed — and rank fourth globally while trailing the closed frontier on factual reliability. The gap isn’t architectural. Moonshot solved the architecture problem. The gap is in what the model was trained to be: a coding specialist at the outer edge of open-weight scale. That’s a real achievement. It’s also a real constraint. K3 leads Program Bench. Claude Fable 5 hallucinates at 16%. They’re not competing on the same surface.

AI-generated editorial illustration · TemperatureZero · July 17, 2026