Qwen3.8 on four Blackwell cards: twice the speed, three times the room

Two machines carry four NVIDIA RTX PRO 4000 cards with 24 GB each, Blackwell, plus a third machine with one RTX PRO 4500 with 32 GB for the night. Kubernetes with Flux, everything from a git repo. On the big machines runs Qwen3.8-27B: 27 billion parameters, weights in NVFP4, cut into four slices across the cards, served with vLLM, with a context window of 512k tokens stretched to twice its trained length by YaRN. In front of it sits a LiteLLM proxy: several deployments behind one model name, and one conversation lands, as far as it can, on the same container. The reason is the prefix cache, which lives per container: the same request on the other machine pays the full prefill again, measured at 12.5 seconds instead of 3.7 for the same prompt. The model carries the day: assistance, code, long documents, cron jobs. The commercial model has been the fallback only since spring.

Sunday morning at seven was the first time it fell on me that this stops running smoothly at large context. Short questions took five seconds, long tasks half a minute, and during the early shift, with fourteen parallel requests queued, some of them gave up after a quarter of an hour. The measurements showed where it stuck:

The third line is the clue. A card that computes draws its 145 watts. A card waiting for the other three draws half, and the utilization gauge still says 100 percent, because to the gauge waiting is work. Four cards keeping each other waiting, and a single card of the next size up was faster in answer speed than all four together. That is not a compute problem, that is a communication problem. Two days of measuring later, same hardware, no card swap: almost twice the speed and five conversations in the window instead of one and a half. The way there took three stages, five mistakes of my own, and at the end a test in the upstream project that fails on every card of this generation without anyone seeing it.

If you read the last post: it was about how we got to half a megatoken of context and what that costs. This one starts where that one ended: with a context window that was big enough, and a machine that no longer ran smoothly underneath it.

Sunday: let the four cards talk

Reference tool first, own opinion second. The standard bandwidth test between the cards said 7.6 GB/s, and the cause was in the same log: all cards hang off the processor with no switch in between, and the communication library considers that topology too far for direct card-to-card transfers. It routes everything through main memory. One environment variable allows the direct path, and the same test then said 40.1 GB/s. Factor five, one line.

Then the engine onto the new version with a fused decode kernel for exactly this model family (which on its own was slower at first, 73 instead of 88, an effect I still cannot explain cleanly), then a different attention kernel and speculative decoding measured properly, with depth four as the best step. Finally async scheduling, plus 15 percent, for free. Each stage on its own, each against the same prompts (P2P is the direct card path, MTP the speculative decoding):

The last line is the catch. The faster attention kernel can only store context at full width, and that cost 40 percent of the room. So on Sunday evening I had a machine that answers twice as fast and holds fewer conversations at once than before. Twelve concurrent real requests still went through without preemption, good enough for daily work. But it was a trade, not a gain, and Monday had exactly one job: get the room back without giving the speed away again.

What the terms mean

Context storage: the compartment where a model keeps what it has understood of the conversation so far, while it writes. A bigger compartment means several long conversations run at the same time instead of queueing. The logs call it KV cache, so that is the word I use from here on.

Compact storage: a space-saving tier for exactly that compartment. The known tiers are called fp8 and nvfp4, after the number of bits per value. The model’s weights have been in NVFP4 for a while; what is new is that the KV cache now lives in that format too. I use the names the way they appear in the logs.

The numbers

Measured on September 8, identical requests on the same machine, only the storage tier changed. The table shows the compartment in tokens and how many sessions at the full window (524,288 tokens, half a bookshelf per conversation) fit next to each other:

On the small 32 GB node the same picture at smaller scale: 239,693 to 387,658 tokens, so 1.69 parallel sessions in a 224k window instead of 1.04.

Answer speed holds: 124 to 126 tokens per second on the big machines, against 129 with fp8. Room has tripled since Sunday morning, speed has almost doubled, and having both at once was not on the table on Sunday evening.

Full context stays correct: at 153k tokens of prompt the marker sits where it belongs and is quoted exactly, in every run, on all three machines. That is the acceptance test that decides over “it just fits more”.

Why halved memory is not doubled capacity

Qwen3.8 is a hybrid: 16 of its 64 layers use the regular attention layout, the other 48 (Gated DeltaNet) carry a fixed-size state store per conversation. That state store stays at full width; the compact tier only applies to the first part. Add one small scaling value per block that is not compressed. That is why you land at +62 to +73 percent instead of +100. My calculation from the evening before had applied the halving to the whole compartment, and that was simply too generous.

For the small node that means concretely: 512k does not fit, by a measurable margin. The engine says at startup the window needs 10.07 GiB, after the weights 7.99 GiB are left. One fifth missing. That is not a tuning question, that is the card.

Monday: one day, 27 commits

Monday is in the ops repo, and the commit timestamps tell it more honestly than I could from memory. Three stages.

Stage one, 04:34 to 06:18. Overnight, the intermediate tier fp8 on the big machines, measured against the previous state: pool +93 percent, 4966 requests in one hour without a single failure (e56ef7de). Same change on the small card, no gain there, so back again (1aa87871). That is the point where a config for one card class does not automatically hold for the other.

The storage tier is tied to the attention backend, and that crossed our path twice. The fast attention kernel from Sunday refuses fp8 in the cache, with a finished hint in the log: the format wants newer attention kernels than this card generation has, “FP8 KV cache requires FA3 on SM90 or FA4 on SM100”. The other two backends, FlashInfer and Triton, accept it without fuss - the card has the fp8 tensor cores, the gap lives in the attention kernels, not on the board. At the end of August the combination fp8 plus FlashInfer passed the full acceptance for the first time on the older engine; with the move to vLLM 0.28 it sits on all three machines and still holds today.

Stage two, 07:39 to 08:58. The question behind the second step: the weights have been NVFP4 since spring, the cards know the format, why not the cache as well? Upstream this was open: enabling this storage tier for our card generation sits in one pull request, the write-path fix in a second, while the big data-center NVIDIA cards have had it for a while. On the big machines nvfp4 buys another +73 percent of pool over fp8, on the small card +62, measured in the table below. The attempt to enable the compact tier with a seven-line patch from an open pull request: startup passes, the first real request dies on an assertion in the decode path (d5f0770a). Then the custom image, on the container’s release version, with four patches from a third-party repo (MIT, attribution included) and a kernel overlay for our card generation. Three builds until the first green one: the first lost files between two build steps (701655dd), the second copied symlinks instead of contents (08e198e8). The Dockerfile checks itself since then, one marker per stage:

OVERLAY-BUILT vllm_sm12x_nvfp4kv.so
torch.ops.vllm_sm12x.reshape_and_cache_nvfp4 registered
NVFP4KV-SM120 IMAGE OK (vllm 0.28.0)

Stage three, 10:06 to 12:14. Staggered onto the three machines, each with the same acceptance run (277b7a7d, 513ae208, 1fdf8129): marker in the log, pool, answer speed, marker sentence at 51k, 107k and 153k. Only when one machine had passed everything did the next one get its turn. In between, the attempt to lift the small card to the same window as the big ones: rejected by the engine (1d502b7a), then a smaller window that produced garbage at 153k, and back to the proven state (afbb6c49). That bug is still open; so far I know it exists, not why.

Five mistakes of my own

From two days, the five I built myself. The fifth is further down, with the test.

On Sunday a second line for speculative decoding sat at the end of the start command, a leftover. The engine takes the last one, so all three steps of my comparison ran at the same depth, and I had concluded “does nothing” from that. Wrong, because of one line I had not read. Since that afternoon the acceptance script prints the effective depth from the engine log before a measurement counts.

A comment block inside the container’s startup command cut the line continuation. After the first rollout nine of twenty-two start options were actually in effect, the rest sat quietly on defaults. During acceptance two answers came out shorter than usual, and one look at the process command line showed it. Since then the acceptance script starts with: count the options, look for comments in the continuation, measure nothing before that.

The evening before, I had estimated the compartment on the small card at about 560k tokens and forgot that the state store stays at full width. Measured: 388k. An estimate that gets treated like a measurement is the most expensive error in this list, because nobody contradicts it.

And during the final acceptance the streaming scheduler pulled the model off the card mid-test because the stream went live. The container was gone, so “marker not found” was not a test failure but a missing test subject. Before measuring on that node I now check whether a stream is running. Banal lesson, still cost a full run. The run after that is the one that counts.

What runs now

Since that run the big machines carry the whole fleet: five sessions per node, 512k window, around two tenths of a second for short questions. The small node takes the night while the others sleep, with its 224k window and two sessions. One look at the last hour’s log: zero restarts, zero preemptions, and through the proxy sixteen of sixteen answers came back complete, from all three machines.

Upstream, the topic is open: enabling this storage tier for our card generation sits in one open pull request, the fix for the write path in a second. My image puts both onto the container’s release version. Once those upstream pieces are merged, the custom image goes away and the config stays. For that to happen the project has to know our numbers, and handing them back is where mistake number five happened.

The test nobody ran

Before handing the numbers back to the upstream project I wanted to see the write path red and green with the project’s own test: red with the original kernel, green with the fix. The test exists, it is named the way you would search for it, and it ran in four seconds inside the container that was answering my requests at the time. Red, as expected. Just in the wrong place:

Mismatched elements: 989 / 10752 (9.2%)
Greatest absolute difference: nan at index (7, 6, 48)

That was the check on the first of the two memory regions, not the second, where the known bug lives. The kernel stores the scale values of the first region linearly, on every card generation. The test reference unscrambles both regions as if they were stored interleaved, and so compares the first one against a shuffled template. The test is red on every card of this generation, with or without the known bug, with or without the fix.

And nobody sees it because nobody runs it: it skips itself below this card generation, and the one CI job that runs on such a card excludes exactly this directory.

I fixed the reference, in three test files, without touching the kernel, and attached the proof as five runs from the container: old reference red in the wrong place, new reference with the original kernel red in the right place, new reference with the fix green. That is the pull request. The measurements from above went as comments under the two open pull requests that carry the actual fix. What I do not have is in there too: a data-center-class card to cross-check on.

What I take from it

Two days in which every number I had estimated beforehand was wrong, and every number I measured held. Sunday evening’s estimate of the cache on the small card, the assumption that speculative decoding brought nothing, this noon’s guess that the upstream test was blind rather than red. All three sounded plausible, all three did not survive a measurement.

The model now answers twice as fast and holds five conversations instead of one and a half. That is the result. The method that got it there was duller: the reference tool first, then one variable per step, then the acceptance run with the marker at 153k tokens, every time, before anything counted as done. Without the marker I would have a small card today with a bigger window and garbage past 150k tokens, and nobody would have noticed until a long conversation tipped over.