A Million Tokens, One Card

A companion to “One Card, One Engine.” The engine boots exactly one format. This is the second file, what it cost, and what this rig’s own tasks say about it.

The one-format engine

One Card, One Engine ended where the container format begins. The engine, NInfer, boots one format, one file, one registration. That is not a limitation anybody bolted on. It is most of why the thing is fast.

What I wanted was a 27B-class model that could hold a million-token window and a vision tower on one card at the same time. The ternary build of Bonsai 2, a Qwen 3.8-lineage model from PrismML, offers that, because ternary weights are small.

What it did not offer was speed. The variant existed only as GGUF and MLX, and while a general-purpose runner loads that checkpoint without complaint, it does not get the engine’s speculative decoding. DFlash2 is most of why this rig is quick. So the compact model was runnable and slow, which on this rig is close to the same thing as absent.

So I built the container.

What it is

One file in the engine’s own format: the ternary language weights, the DFlash2 adapter stack, the MTP head, and the vision tower. 1,193 objects, 9,520,051,456 bytes, and byte for byte the same size as the official container of the same model. That last fact earns its own section later.

None of the model is mine. PrismML built the ternary base. OS-Software did the decensoring, baked into the ternary codes themselves. The Qwen team supplied the base model, the vision tower and the tokenizer. The WaveCut engine line supplied the conversion recipe. ProCreations supplied the MTP head and the DFlash2 adapters. Apache-2.0 the whole way down. The model card carries the full provenance and the checksums if you want to check the work.

What I did was package it and measure it.

One thing worth stating plainly, because a benchmark table can imply otherwise: it is not in service. Nothing on this rig runs it today. It is kept ready, a local 1M-window option for whenever a workload needs one.

Why the size is the argument

The weights in this container come to 7.99 GiB. The runtime footprint at a million-token window is 19.1 GiB, and about 10 GiB of that is key-value cache, with the vision tower in the same file.

That is what the ternary format buys: weights small enough that the space left over goes to context. Orca 27B, the build this rig already serves, carries 315,000 tokens of window. This one carries a million, vision included, on the same card. The size is the argument. The de-refusal is the part I could measure.

What de-refusing cost

The decensoring touched 34 of 851 matrices. That is about four percent of the weights, and it was not a re-training or a re-alignment. It was a surgical edit, baked into the ternary codes.

The card measures what that cost, against the official container of the same model, on the same harness.

Quality went from 5.855 to 5.876 perplexity, up about a third of a percent. Draft acceptance went from roughly 25 to 32 percent down to roughly 25 to 30. Decode ran 296 to 343 tokens a second on the official artifact, and 325 to 327 on this one.

Put plainly: de-refusing this model costs about a third of a percent of perplexity and nothing measurable in throughput.

Why that is worth a page rather than a changelog line: before this container the variant had no artifact in the engine’s format, so this cost had not been measured there. This is that measurement.

The card keeps its own hedges, and so do I. The artifact is smoke-tested, not benchmarked. And its dependable retrieval window is narrower than its configured one: somewhere between half a million and seven hundred thousand tokens, not the full million it will accept.

What this rig’s own tasks say

Public scores are a game with a known strategy. Publish a number and somebody optimizes for the number. So the container went through the battery this rig actually runs, on the same tasks as Orca 27B, the build the rig already serves, one model loaded at a time, with the weights checked in memory between runs.

Twelve tasks, three runs each. Both models answered every one correctly.

Then a code review: a small file with five defects seeded into it, two runs per model. Orca 27B found all five in both runs, with tight line citations. The container missed the dead branch both times, and when it found the others it sometimes pointed a few lines off, up to four.

Two runs is a small sample, so I read that as a signal rather than a verdict. It is still the honest shape of the thing: whatever this container gives up, it gives up in review work where a quiet dead branch matters.

Speed is the other half.

The container decodes at a median of 535 tokens a second against Orca 27B’s 226, which is the number that made me write the piece. At 25K depth it holds 492 against 215, and on the deep runs 346 against 180. First token on a short prompt lands in about a tenth of a second, three times quicker than Orca 27B. Prefill is the trade: Orca 27B holds its rate better into long prompts, 7,813 tokens a second against 6,585 at the same 25K depth, and it gets to the first token of a long prompt far sooner.

Depth labels in that table are the prompts the runs actually received, with the benchmark’s targets in brackets. The deep rows were not even the same size: the container was fed 164,000 tokens to push its window further out, and Orca 27B 82,000. Longer prompt, longer wait. It is not like for like, and I would rather say so than let a table imply otherwise.

Both models retrieved. Needle recall was three of three in every depth run for both.

The bounds, as bounds. One session, one run set, default sampling, engine defaults for the key and draft settings, two review runs per model. Vision quality and concurrency headroom were not measured. And there are two sets of numbers in this piece: the card’s, measured on its harness, and mine, measured on this rig’s defaults. Where they differ, that is the reason.

One trick worth stealing

The container is byte for byte the same size as the official artifact. Same container, different weights.

If you convert models, that gives you a pre-flight check for free. Convert a variant into a reference container, then compare the output size to the reference. A different size means the conversion is wrong before you ever load it, which is the cheapest failure you will ever get.

The error worth keeping

An early smoke test suggested draft acceptance had collapsed from about 60 percent to about 20. It had not. The two runs had compared different content. Measured on the same content, the decensoring cost essentially nothing in throughput.

The same shape of lesson ran through the previous piece. A number is only as good as the comparison behind it, and the comparison is the part you have to check.

The card is candid about its limits, and I would rather quote it than smooth it. It is smoke-tested, not benchmarked. Retrieval degrades at the far end of the window, inherited from the base model. And this line, which matters more than the others: this model is deliberately decensored and will answer requests the official model refuses. Deployment needs its own access controls and policies, and nothing in the weights enforces them.

What the shelf looks like now

The variant that ran slowly now runs at engine speed, carrying a million tokens and a vision tower on a single card. The de-refusal that got it there costs about a third of a percent of perplexity and nothing measurable in throughput, and on this rig’s own tasks it generates about two and a half times faster, giving up something real on review work.

It is still not in service. It exists so the option does.

Packaging and measurement, nothing more. The model belongs to the people who built and edited it. The container belongs to whoever needs it next.