One Card, One Engine: more than doubling local inference speed with NInfer

A companion to “Inside the Quiet Fleet.” What it took to retire llama.cpp, and why the new rig is mostly made of contracts.

The old rig

The old rig died of two things: a cleanup, and a ceiling.

It ran on llama.cpp, behind a controller I wrote myself. The trick it pulled was convenience: one endpoint, any model name I liked, one model resident at a time, swapped on demand. It was a good trick, and it held up longer than it had any right to.

Then a disk cleanup deleted the model registry. Most of the weights were gone at once, and half the aliases pointed at files that no longer existed. That was the cleanup. The ceiling was quieter. Decode speeds topped out around 95 tokens per second, and every new capability I wanted, whether vision or longer context or speculative decoding, meant another permutation of aliases and flag templates. Each permutation was a small promise to keep, and the promises were starting to outnumber the models.

So I went looking for a rig that only wanted to do one shape of work.

The chase

NInfer is a from-scratch engine built for exactly one GPU. Not one class of GPUs. One card: an RTX 5090 with 32 GB of VRAM. That narrowness is the whole point. When you only have to be fast on a single piece of silicon, you can spend your complication budget on speed instead of portability.

Models arrive as a single file. One container holds the weights in an NVFP4 quantization, the draft head for speculative decoding, the vision tower, and the metadata that describes them. There is no pairing file to lose, no chat template to patch, no folder of flags that must agree with each other. One file, one registration, and the engine boots it.

The model I started with is Orca 27B, a Qwen 3.8 27B build. The project’s published numbers are around 210 tokens per second in its multi-token-prediction mode, and up to around 356 with its newer drafter. Against the old rig’s 95, that is a little over twice as fast, and up to nearly four times. Worth chasing.

One thing to say plainly, because it shapes everything below: I did not take this engine off the shelf. I build and adapt it to fit this rig. The Windows port it runs on is my work, published in a public fork. The controller wrapped around it is mine. The profiles are tuned for this exact card. That ownership is the reason the rest of this story has lessons in it.

The endpoint

Around the engine sits a small controller, rebuilt from the pattern the old rig used. It publishes one port to the network. Behind it, a single model lives at a time and serves two requests at once. The controller swaps children on demand. A swap costs about six seconds, cheap enough that nobody has to think about it.

Four profiles rotate through it, matched to the card’s memory: two speculation modes with vision, two without. Legacy alias names from the old stack remap to the nearest profile, and the engine echoes back whatever model string a client sends. Zero client reconfiguration. When the card needs to render instead of think, a single call frees all of it.

Porting YaRN to Windows

The new engine had one gap that mattered: context. The Windows build capped the context window at the model’s native 262,144 tokens. Another fork of the same engine had YaRN, a rotary-scaling technique that stretches a model’s usable context past what it was trained on, but that fork was Linux-only. This rig runs Windows.

So I ported it. Three commits, about 280 lines across 16 files. Fifteen of the sixteen files took the patch cleanly. The last one needed two hunks hand-resolved against newer command-line code, and one build pin to settle a toolset mismatch. The port is public.

Two details make this worth reading instead of just reporting. The first is flash attention. Coming from llama.cpp habits, the natural question is whether the engine supports it, and the answer is that there is no flag because there is no non-flash path. The kernels implement it by hand: attention streams through in 64-token tiles under an online softmax, so the full score matrix never exists in memory. That is part of why decode still holds around 200 tokens per second past 200,000 tokens of context.

The second is verification. The flag refuses to lie. Ask for a 393,216-token context without the scaling factor and it is refused. Hand it the factor and it is accepted. Then a needle test at roughly 309,000 tokens, well past the native wall, retrieved both needles planted earlier in the text. That is the standard I want for the rest of this piece. Not “it should work.” Here is the run.

Trust, but parse the container

Before trusting the weights, I checked what they were.

Three public repositories claimed the same checkpoint, and their filenames disagreed. So instead of believing any of them, I parsed the container format directly: an 8-byte magic number, a length, then a JSON directory describing every object inside. The format reads over HTTP range requests, which means you can inspect a multi-gigabyte model before committing to the download.

The base image held 1,124 objects: weights, draft head, vision tower. One candidate was a strict superset at 1,190 objects, and the extra 66 were a newer speculative-decoding upgrade that a flag unlocks without any conversion. A third was an older quantization recipe wearing a similar name. The two files I kept were verified byte-exact against the host’s storage records.

Filenames are marketing. The container is the truth.

The drafter, and what it measured

DFlash2 deserves its own explanation, because it is not a model. It is a drafter: a five-layer miniature network that proposes candidate tokens in whole blocks, one masked pass at a time. The main model still verifies and authorizes every token that actually gets emitted, so correctness per token is structural, not a hope.

The comparison that decided it was measured, not vibes: MTP against DFlash2, same six prompts, same seed, one request each, in a shape of work that matches what this rig is for.

Every one of the six improved. The mean moved from about 192 to about 256 tokens per second, a third faster.

Story was the toughest case. Its published acceptance rate is only about 17%, and it still improved, because block drafting amortizes the loss across each round.

One honest caveat, from upstream’s own benchmarks: verification guarantees each token is correct, not that the trajectory makes sense. A pathological repetition loop showed up on one long-reasoning fixture.

The ledger that actually binds

The card has 32 GB. The desktop takes about 3 of them. The engine asks for its reservation plus a gigabyte before it will even start, and that startup check is the real gate. In between sits the KV cache, about 18.9 KiB per token at this quantization, which means context is something you budget, not something you have.

It is also a budget for two requests.

Every profile keeps more than two gigabytes of headroom at two concurrent requests, and adding the second request costs less than a gigabyte.

Two surprises came from measuring instead of assuming. Vision costs about 0.6 GiB, far less than the log lines suggest. The alarming numbers are internal caps rather than real allocations, and trimming them changed the reservation by zero bytes. And the rotary scaling factor only pays off above the native window. Below it, native is strictly better, so a small factor stays in the config purely for headroom.

First week, three wounds

All three wounds are contract lessons, which is this piece’s thesis arriving on schedule.

The first real client died on its first request. A remote coding agent, hitting the new endpoint, failed with the kind of error that sends me to the logs. Its traffic was arriving fine. The child process held the verdict: the agent asked for a reasoning effort level the engine’s embedded template does not support, and the template rejects unsupported levels at request-prepare time. The old stack never saw this because its template had been hand-patched long ago to quietly remap those levels. A patched template is a contract adapter by accident, and I had been leaning on it without knowing it. The fix went into the router as a normalization step before forwarding. No client changed.

The second was an address mistake with a subtle tax. A client had been pointed at the controller’s internal swap port, which is loopback-only by design and respawned with every swap. On the client’s side, that address resolved to its own loopback, which refused, retried, and fell back. Every call still worked, just not through the fast lane, and the rack looked guilty the whole time. The rules that came out of it: clients only ever touch the controller’s public port, and a fast lane is an alias on that port, never an internal one. Topology is part of the contract.

The third is subtler, and it would have aged badly in silence. Harnesses learn a model’s context window by reading the model listing. The engine’s own listing clamps some model names to a wrong, smaller number. Rather than patch the engine, the router now serves its own listing, where a single field drives both the launch flag and the advertised number. They cannot drift apart, because they are the same value. The controller is not a pipe. It is where the external contract is enforced.

What I deliberately didn’t do

An honest ledger has a second column: what I chose not to build.

There is no authentication on the controller port. The local network is trusted, so that is a private-network tradeoff with a stated cost, not a virtue. I did not combine the new drafter with the extended context, because the combination is unvalidated both upstream and in this port, and it would not fit in memory anyway. And I kept the rollback: the old files and roughly 35 GB of old models are still on disk, because a migration I cannot undo is not a migration. It is a wager.

The point

Here is what I would tell you to write down, after all of it.

Whatever your stack quietly promises, find the place where each promise lives, and write it down. Not in a wiki nobody reads. In the place that enforces it. A promise that lives in someone’s memory, or in a patched template, is a promise you do not own.

The rig is one card and one engine. The rest is contracts I now own on purpose.

The engine fork and the rig’s configuration are both public: ninfer-5090-windows-yarn and ninfer-orca-stack.