Bonsai 27B Fits in 4 GB. Post-Training Quantization Never Could.

Yesterday, PrismML released Bonsai 27B: two builds of Qwen3.6 27B that fit in 3.9 GB and 5.9 GB respectively, run at 11 tokens per second on an iPhone 17 Pro, and retain over 90% and 94% of the full-precision model’s benchmark performance. The headline most coverage led with was ’27B model runs on a phone.’ That’s true. The more durable story is about the method that made it possible — and why the gap between the benchmark averages and the benchmark breakdown is where the technical content lives.

Bonsai 27B is a 27.8-billion-parameter multimodal model, released July 14 under Apache 2.0. The two variants — a 1-bit binary build at 1.125 bits per weight, and a 1.58-bit ternary build with weights constrained to {-1, 0, +1} — are not the result of taking a trained full-precision model and compressing it afterward. They are the result of training with binary or ternary constraints baked in from the start, across every component of the architecture: embeddings, attention layers, MLP blocks, the LM head. No higher-precision fallback. That distinction is the entire technical story.

The Floor That Post-Training Quantization Can’t Cross

Post-training quantization (PTQ) works by rounding the weights of a trained high-precision model to fewer bits. At 8-bit, the rounding error is small enough that models retain near-perfect quality. At 4-bit, you lose a few benchmark points but the model remains functional — most current on-device models ship at 4-bit, and most work well. At 2-bit, PTQ starts cutting into capabilities that required high-resolution weight representations to maintain. At 1-bit binary, where each weight is +1 or -1 and nothing else, PTQ essentially destroys the model. You’re replacing a continuous parameter space with two discrete values per weight; the quantization noise overwhelms the signal that made the original model capable.

The problem is also visible in what the compression does to individual weights. A transformer trained in FP16 develops weight distributions that are roughly Gaussian with many values near zero — the superposition hypothesis suggests that features are encoded partly in small-magnitude weights that can be negative or positive but are rarely extreme. PTQ at 1-bit maps every weight to +/-1: a weight of 0.73 becomes 1.0, a weight of -0.12 becomes -1.0, a weight of 0.001 becomes 1.0. The fine-grained structure in the near-zero region — which carries meaningful representational content — is obliterated. There’s no training signal that prepared the model for this resolution.

The alternative is quantization-aware training (QAT): the model trains with the target bit-width constraint applied throughout, so the weights never exist at high precision in the first place. The model learns to represent knowledge in binary or ternary space rather than being forced into it afterward. Microsoft Research’s BitNet b1.58 paper, published in February 2024, demonstrated that ternary QAT at standard transformer scales could match full-precision perplexity and downstream task performance while reducing latency, memory, and energy substantially. The core claim: restricting weights to {-1, 0, +1} during training doesn’t degrade the model the way PTQ does, because the model finds an optimization basin that happens to be efficiently representable at ternary precision. What BitNet b1.58 hadn’t done — or hadn’t published — was apply this at 27B parameters with multimodal vision support and 262K context targeting agentic workloads. That’s what PrismML did, trained on Google v5 TPUs with the constraint fully embedded, spanning every layer of the architecture without any higher-precision fallback.

One implementation detail is load-bearing. Bonsai’s model card specifies binary g128 format: individual weights are +/-1, but each group of 128 weights carries an FP16 scaling factor. The effective computation is binary weight times group scale. This preserves magnitude information at the group level — without it, you’d lose the ability to represent different-valued activations across weight groups, and the quality floor would be far lower. The 1.125 bits per weight figure reflects the weight values alone; the group scales are additional overhead but small relative to the full weight matrix. The 3.9 GB footprint is 14.2x smaller than the 54 GB FP16 baseline, and the reduction is structural rather than compressive.

Reading the Benchmark Breakdown

The 1-bit Bonsai variant scores 76.11 average against an 85.07 baseline across 15 benchmarks — 89.5% retention. The ternary variant scores 80.5, which is 94.6% of the full-precision model. Those headline numbers average across categories that behave very differently under extreme quantization, and the distribution matters more than the single-number summaries.

Math (GSM8K, MATH-500, AIME25, AIME26): 91.66 on the 1-bit variant. Coding (HumanEval+, MBPP+, LiveCodeBench): 81.88. Knowledge and reasoning (MMLU-Redux, MuSR): 73.39. These three categories — covering most of what developers would actually deploy on-device AI for — are within 8 to 12 points of the full-precision baseline. Notably, the model card reports that conventional 2-bit PTQ variants drop to 57.5 on AIME26, while 1-bit Bonsai holds above 87 on that benchmark — the hardest math eval in the suite and the one where degradation from training-level compression would be most expected. The ternary variant closes the gap further, with ternary math above 94 and ternary coding above 87.

Vision (MMMU-Pro, OCR Bench v2): 59.57 on the 1-bit variant — roughly 25 points below the FP16 baseline. Instruction following (IFEval, IFBench): 65.74. Agentic and tool calling (BFCL v3, tau-squared-Bench): 66.03. These categories are where binary quantization concentrates its cost. They demand fine-grained spatial reasoning, multi-step instruction parsing across nested steps, and reliable state tracking across external function calls — capabilities that depend on weight resolution in ways that discrete symbol manipulation does not. The 10% aggregate loss isn’t distributed evenly across the model; it lands almost entirely on these three categories.

The HN community found this in practice. Multiple developers reported tool-calling failures, reasoning loops, and degraded vision handling under real workloads. One found the 1-bit Bonsai performing worse than Gemma 4 12B on SQL tasks — a 12B model outperforming a 27B-class model on a specific coding benchmark. These aren’t surprises given the 66.03 agentic score. Developers running into tool-calling failures on the 1-bit variant are encountering the benchmark, not discovering something the benchmarks missed.

Worth noting: the 15-benchmark suite is PrismML’s own evaluation, not a submission to LMSYS Chatbot Arena or any independent leaderboard. The benchmark categories are standard and well-scoped — AIME is hard enough that sandbagging results would be unusual, and MMMU-Pro is a widely-used industry eval. But first-party benchmark results and independently-replicated benchmark results are different claims, and only one of them currently exists for Bonsai 27B.

What 4 GB on a Phone Actually Unlocks

Strip away the caveats and a clear use case remains. A 27B-class model that scores 91.66 on math and 81.88 on coding, runs entirely offline, costs zero per inference, and fits the memory budget of current flagship phones has a straightforward deployment profile. Offline coding assistants, math tutors, structured reasoning tools for jurisdictions with data residency requirements, air-gapped enterprise tooling — these are workloads where 1-bit Bonsai is a viable production option rather than a capability demo. The ternary variant at 5.9 GB narrows the quality gap further and fits comfortably on laptop memory budgets; at 87 tokens per second on an M5 Max, it’s fast enough for real interactive use.

For vision, complex instruction following, and agentic tool-calling, the benchmark profile is explicit: this is not the model for those workloads yet. The three categories where binary quantization extracts the highest cost are also the three where on-device AI needs to be most reliable if it’s going to replace cloud inference in production systems. That’s not a failure of execution — it’s the inherent tradeoff of the bit-width, and PrismML published the numbers rather than averaging over them. The path to closing it is either the ternary variant, which recovers several points in each degraded category, or further maturation of QAT methodology at larger training compute budgets.

Concurrent independent research suggests this space is moving quickly. An arXiv paper published this month, Variable Bit-width Quantization by Hamish Ogilvy, demonstrates a different QAT approach: each group of 64 weights learns its own bit-width from {1, 2, 4, 8} bits during training, with the model autonomously discovering that 69% of weight groups can operate at 1 bit while early MLP blocks need ~2.5 bits. Different method, same direction. The threshold at which sub-2-bit QAT produces viable models is being crossed from multiple directions simultaneously; the question for the field is not whether the approach works but how quickly the capability floor rises as training methodology matures and model families are designed around these constraints from architecture inception rather than retrofitted to them.

What PrismML established with Bonsai 27B is the method’s viability at 27B scale with multimodal capability, trained end-to-end at 1.125 bits per weight, retaining 91% of math capability and 89.5% overall. The press angle is the phone. The technical angle is what happens when you stop trying to compress models down to binary and start building them there. Those are different problems — and only one of them has a floor.

AI-generated editorial illustration · TemperatureZero · July 15, 2026