Nvidia Shipped Memory-Safe GPU Kernels. One Track Has Adopters. One Doesn’t.

On September 8, Nvidia announced native Rust support for GPU kernels. The announcement was framed as a unified breakthrough — memory-safe GPU programming, arriving at last, via two complementary tracks. What the launch blog disclosed, buried in a secondary clause, was that cutile-rs was already being used by HuggingFace’s Grout inference engine and mistral.rs. Not adopted in response to the announcement: already running. HuggingFace’s academic paper on cuTile Rust was submitted to arXiv in June 2026, three months before the official launch. The inference community didn’t move fast because the announcement was compelling. They moved first because the tool solved a problem they’d been working around for years.

The problem is this: the GPU kernel is the one place where Rust’s ownership model has historically stopped. You could launch kernels from Rust. The kernel itself had to be written in something else, usually CUDA C++, which has no mechanism for preventing data races at compile time. A shared memory aliasing bug in a fast attention kernel can produce silently wrong outputs on some runs and correct outputs on others, depending on hardware scheduling. You don’t always know it’s wrong. That’s the failure mode that Rust’s type system is designed to prevent everywhere except, until now, inside the GPU.

Two Tracks, Two Different Deals

Nvidia shipped two distinct tools under the CUDA Rust umbrella. The first is cuda-oxide, a custom rustc codegen backend that routes kernel functions through Rust MIR, the Pliron IR framework, and LLVM down to PTX. It preserves the SIMT programming model — Single Instruction, Multiple Threads — where you write the kernel body as if from the perspective of one thread and the hardware fans out across the grid. It adds safety primitives: DisjointSlice<T> enforces exclusive per-thread access to output buffers, and a #[launch_contract] macro validates indexing at launch time. It requires a pinned nightly toolchain (nightly-2026-04-03, specifically), a custom LLVM build, and Linux. It is early alpha. It has no publicly listed adopters.

The second tool is cutile-rs, and it is a different kind of thing entirely. It runs on stable Rust 1.89 with no custom toolchain and no custom LLVM. A #[cutile::module] macro embeds the kernel AST in the host binary and JIT-compiles it through CUDA Tile IR at first use. The programming model is not SIMT. You write the kernel body as a single logical thread operating over a sub-tensor of data, call .partition([128]), and the compiler handles grid geometry. Exclusive ownership is guaranteed through tensor partitioning — each tile block owns disjoint output regions by construction, and Rust’s borrow checker enforces it. There is no unsafe needed for the common case. cutile-rs is published on crates.io. HuggingFace’s Grout inference engine and mistral.rs have adopted it.

That asymmetry — one tool in early alpha with no adopters, one on stable Rust with two significant adopters inside a week — is the story Nvidia’s announcement diplomatically obscured. They’re not shipping one thing. They’re shipping a bifurcated programming model, with different audiences, different maturity levels, and different safety guarantees.

What the Paper Shows

The academic foundation for cutile-rs is a June 2026 paper from the HuggingFace research team, “Fearless Concurrency on the GPU,” which describes three safety mechanisms: partitioned mutable tensors prevent data races by ensuring each tile owns disjoint output regions; token-ordered memory operations preserve sequential semantics for mutable references; branded partition indices verify multi-output schedule safety at compile time without runtime checks. The paper also builds Grout as a reference implementation — a Qwen3 inference engine running entirely on cuTile Rust.

The performance results are what matter here. On a B200, Grout achieves 2.07 PFlop/s on GEMM — 96.4% of cuBLAS, within 0.3% of unsafe Rust. Element-wise operations hit 7.02 TB/s, matching cuBLAS. For end-to-end inference on Qwen3-32B at batch-1 decode, Grout reaches 80.1 tokens/second on a B200. vLLM hits 77.5. SGLang hits 76.5. The safe-by-default code is competitive with the serving stacks the industry standardized on.

Two caveats are worth naming, both present in the paper. First, Grout is framed explicitly as a “testbed” — research software with a benchmark harness, not a production serving stack. It supports Qwen3-4B and Qwen3-32B on RTX 5090 and B200. The diversity of workloads, models, and quantization schemes that production deployments handle is not there yet. Second, the benchmark is decode-only at batch-1 — the scenario that maximizes per-token throughput and shows the Tile abstraction at its best. Prefill-heavy workloads or batched serving patterns are different terrain.

Neither caveat undermines the result. What the paper demonstrates is that the Tile abstraction does not sacrifice throughput for safety on the workloads it was designed for. The performance parity with cuBLAS is close enough that “safe-by-default costs you performance” is no longer a defensible objection to cutile-rs for standard inference serving.

The Shared Memory Problem

The SIMT track is where the tension lives. Shared memory — the high-bandwidth, explicitly managed scratchpad on each streaming multiprocessor — is the foundation of every fast GPU kernel that goes beyond standard matmul. FlashAttention’s performance comes from tiling attention computation to fit in shared memory and avoiding redundant HBM reads. Custom quantization kernels (int4, fp8 with custom scaling) use shared memory to stage dequantization. Fused operators that combine matrix multiply and activation use shared memory to avoid writing intermediate results to global memory. These are the kernels that researchers write, iterate on, and publish; they’re the ones that move the performance frontier.

In cuda-oxide, shared memory currently requires unsafe. The launch blog confirms it: shared memory is “the bedrock of fast SIMT kernels” and making it safe is described as future work. Which means the SIMT track, as of September 2026, cannot provide safety guarantees for the single most important optimization surface in GPU kernel research. You get DisjointSlice and #[launch_contract], which prevent some aliasing errors, but the scratchpad itself is still an unsafe block. CUDA C++ doesn’t look worse by comparison than it did before the announcement.

This isn’t a criticism of Nvidia’s direction — it’s a description of where the engineering frontier sits. Safe shared memory access in a SIMT model is a genuinely hard problem. The Tile track solves it by taking shared memory out of the programmer’s hands entirely; the compiler owns the memory layout. That’s the trade: you get safety by giving up control. For inference serving, that trade is favorable. For a researcher writing a custom attention variant or a quantization kernel with a non-standard memory layout, the tile abstraction rules them out of the optimization space they care about.

The comparison to Triton is instructive. OpenAI’s Triton, a Python-based GPU kernel DSL, operates at the same level of abstraction as cutile-rs — you write programs over tiles of data rather than individual threads, and the compiler handles shared memory. Triton is widely used in inference and has influenced the architecture of both FlashAttention-3 and many production attention kernels. But it’s Python. The researcher community uses it for rapid iteration; the kernel ends up in a C++ extension for production. cutile-rs offers the same tile abstraction with Rust’s compile-time safety guarantees instead of Python’s runtime flexibility. That’s a genuine improvement for the use case it serves.

The Bifurcation Is the Strategy

Read as a product strategy, the two-track structure is legible. Nvidia has two distinct customer types for GPU programming. The inference community — serving companies, cloud operators, framework engineers — needs reliable, maintainable kernels that don’t silently corrupt activations. They write code once, deploy it widely, and run it billions of times. Memory safety at compile time is directly valuable; the cost of a CUDA bug in production is measured in user-facing failures, not research cycles. cutile-rs is built for them, and it works now.

The research community — ML researchers, kernel engineers at labs, people writing custom attention and quantization kernels — needs control. They need to tell the GPU exactly where things live in shared memory, exactly which threads synchronize on which barriers, exactly how the warp-level instructions land. The safety constraints of the Tile model are constraints on the optimization space. cuda-oxide is theoretically the path for this community, but it’s alpha, requires a custom toolchain, and doesn’t yet cover the part of the stack they care about. In practice, this community is staying in CUDA C++ and watching the SIMT track mature.

Nvidia’s roadmap includes “inter-language interoperability between CUDA Rust, CUDA C++, and CUDA Python” — the path by which cuda-oxide kernels could eventually coexist with C++ kernels in the same binary. That’s the real convergence story. When the SIMT track covers shared memory safely and runs on stable Rust, the argument for CUDA C++ in research kernels gets harder to make. Today, that argument is still easy. The announcement is the beginning of a multi-year transition, not the arrival of it.

What happened this week is that the inference half of GPU programming got a memory-safe Rust stack that delivers competitive throughput and is already being adopted by the projects that matter. That is real and significant. The research half is watching.

AI-generated editorial illustration · TemperatureZero · September 17, 2026