The default moves local

For a few days now my daily work has been running on a model that sits in the basement. The large commercial model that used to carry everything demanding is still wired up, but only for emergencies.

Three months ago I would not have believed that. Not because local models are bad, but because they always failed at one of three points: too slow for a conversation, too forgetful for real tasks, or they could not operate a tool and could not read an image. A model that only writes prose does not replace a workplace.

Those three points hold now. The response speed is beyond what I can read. The memory holds a good half shelf of books per conversation, which is enough for work that runs across a morning. And it operates tools and reads images, which is the difference between a chat window and something you hand work to.

Getting there took four weeks of measuring, and the answer was rarely where I looked for it. For two weeks my notes said a certain memory setting was impossible on this hardware. Well supported, reproduced several times, the software crashed reliably. The setting was possible. I had looked in the wrong place, and the wrong place had answered plausibly.

While cleaning up those notes I counted how often that happened: ten times in four weeks. Always the same pattern, always a source that answers instead of the one that decides. That is the thread running through this post.

Underneath sits the engineering: three models behind one name, every tuning position measured on its own, and a model large enough that it has to be split across two machines. Every number here is my own, the mismeasurements included.

What the terms mean

Five terms, then every table below is readable.

Single request (tok/s): how fast one answer for one person appears. This is what you feel in a chat. 205 tok/s means a 600-word answer in about three seconds.

Aggregate (tok/s): how many tokens the server delivers in total when several requests run at once. It rises with parallelism because the GPU interleaves sequences instead of waiting. This matters when crons, watchdogs and chat all ask at the same time. It does not matter for your own single request.

MTP / speculative decoding: normally every token costs a full pass through the model, which means reading 27 billion parameters out of VRAM. That is bandwidth bound and it is the hard limit. MTP flips it around: a small drafter guesses several tokens ahead and the big model checks them in one pass. Whatever survives is gained. The output is bit-for-bit identical to running without MTP, because the big model verifies every token. No quality loss, by construction.

Acceptance length: how many of the proposed tokens survive per pass. The acceptance rate alone misleads you: with a deeper drafter the rate drops while the average length still goes up. Watch the length.

KV cache: the memory holding the context of a running conversation. A bigger KV pool means more long conversations at once, not faster tokens.

The 27B and the road to TP4

The model: a 27B dense checkpoint in NVFP4 (4-bit weights, a community build, not one I made), tensor parallel 4 across four RTX PRO 4000 cards, 24 GB each and 554 GB/s of VRAM bandwidth per card. The weights split into four ranks of roughly 5.5 GiB, and whatever is left on the card becomes KV cache. This is the production invocation, as it stands in the cluster:

exec vllm serve /model/models/dense-27b-nvfp4 \
  --served-model-name=dense-27b-xnode \
  --host=0.0.0.0 --port=8500 \
  --tensor-parallel-size=4 \
  --distributed-executor-backend=ray \
  --trust-remote-code \
  --gpu-memory-utilization=0.85 \
  --max-model-len=524288 --max-num-seqs=8 \
  --attention-backend=TRITON_ATTN \
  --kv-cache-dtype=fp8_e4m3 \
  --hf-overrides='{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":2.0,"original_max_position_embeddings":262144}}}' \
  --enable-chunked-prefill --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser=<xml-parser> \
  --reasoning-parser=<reasoning-parser> \
  --speculative-config='{"method":"<mtp-method>","num_speculative_tokens":4,"attention_backend":"TRITON_ATTN"}'

Four things in that line are the actual decisions. Everything else is filler.

• --max-model-len=524288 plus --hf-overrides with YaRN factor 2.0. The model declares 262k; the override takes it to 512k. VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 sits next to it as an env, because otherwise vLLM rejects longer values.
• --speculative-config with num_speculative_tokens=4. Depth 4 is the production value and it comes out of the sweep further down.
• --enable-prefix-caching. The cache that the entire load-balancing half of this post revolves around.
• --kv-cache-dtype=fp8_e4m3 together with two separate mentions of TRITON_ATTN. That is the note from my notebook that read “impossible” for two weeks, and it needs its own section.

These are the flags from the vLLM CLI documentation (https://docs.vllm.ai), none of them invented here.

The two weeks when fp8 was impossible

The KV cache sits at 1,519,365 tokens live, concurrency 2.90x at 524k per request, measured on the running process. Against the bf16 state of the same instance (648,858 to 665,573 tokens, depending on the prefill budget) that is a good factor two. I had written off that exact number as unreachable two weeks earlier.

The observation was correct: with --kv-cache-dtype=fp8_e4m3 the engine died reproducibly with illegal memory access, reproduced three times. My explanation was “this card cannot do fp8 KV”. That sounded plausible, because sm120 is a young architecture and kernel gaps are normal there.

The route that actually answers the question is not the capability list, it is asking every backend individually with the real config. vLLM has validate_configuration for that. The result, across all 35 backends:

fp8_e4m3 ACCEPTED by: TRITON_ATTN, FLASHINFER, ROCM_ATTN,
                      ROCM_AITER_UNIFIED_ATTN, CPU_ATTN
fp8_e4m3 REFUSED by:  FLASH_ATTN ['FP8 KV cache requires FA3 on SM90
                      or FA4 on SM100'], FLEX_ATTENTION

Two out of 35 say no, and one of those was the one I had read. The card has fp8 tensor cores. What is missing is an sm120 kernel variant in FlashAttention. That is a gap in a library, not a property of the hardware, and that distinction is the whole difference between “cannot be done” and “can be done, just not that way”.

Which leaves the question of why it crashed anyway, since FLASH_ATTN was not set. vLLM had picked for itself, and it picked FlashInfer: Using FLASHINFER out of potential ['FLASHINFER','TRITON_ATTN']. FlashInfer has no static CUDA graph support, degrades to UNIFORM_SINGLE_TOKEN_DECODE once speculative decoding is on, and then it dies. TRITON_ATTN carries _cudagraph_support = ALWAYS written into the code. So the crash did not come from fp8, it came from the combination of fp8 and a backend choice I never made.

And then the same mistake arrived a second time, in a place I considered settled. --attention-backend=TRITON_ATTN set, crash gone, all good, until batch width 32: illegal memory access in kernels with eagle_ in the name. That is the MTP draft layer, and it had kept choosing FlashInfer, because --attention-backend applies to the main layer only.

At the time I searched envs.py for a matching environment variable, found none and concluded there was no flag for it. So instead of the cause I treated the symptom and capped the batch width. There is a flag. It is not an environment variable, it is a field inside --speculative-config:

"attention_backend":"TRITON_ATTN"

Two lines in the manifest, two weeks of delay, and both times the cause was the same: I had looked in a place that matched my question but not the answer.

--max-num-seqs is the most interesting number in the line: it sits at 8. The isolated throughput measurement says 32 would be better. Going from 8 to 32 lifts aggregate throughput from 613 to 972 tok/s (+46.8 %), and with the mamba-float16 tuning on top, 984. The single request does not move. Production still runs at 8, and I wrote down the reason for that twice before I got it right.

First attempt: with MTP and fp8 KV, batch width 32 crashed reproducibly with illegal memory access in the eagle kernels, while a differential run with MTP off went through clean. Conclusion back then: hard scheduler limit, MTP is worth 1.9× on a single request, so cap the width. That was treating the symptom. The cause was the attention backend of the draft layer, not the width. vllm#54225 shows the same crash with one request of 16 tokens. With TRITON_ATTN on both layers, the crash threshold disappears entirely.

The 8 stays anyway, now for the right reason: with cron plus one or two chat sessions, width 32 is never reached. The cap is no longer a crutch but a guarantee. vLLM never runs more than eight sequences at once, and that is exactly what makes the KV pool per sequence predictable.

And here is the number that made me angry first: 35 tok/s on a single request, measured on the same engine that produced 205 on another day.

That was not the system falling behind. The “single request” was not one. Other requests had been running before the measurement, and the prefix cache counter rose by 600,000 during twelve seconds of idle. A value that comes out identical under every condition is first a suspicion about the measurement, not about the model. Fixed: an idle check is now mandatory, and foreign load aborts the run instead of poisoning it. 205 tok/s without thinking is the honest number. With thinking it is 55, and that too is not a measurement of the model but of how much thinking a prompt draws out.

512k of context, and what YaRN actually costs

The model declares 262k. It reaches 524,288 with YaRN rope at factor 2.0, and the price turned out to be the most interesting number of the whole round: nothing. Before, 8/8 correct on retrieval; after, 8/8; no regression in a single task. The warning in the model card (“may reduce quality on shorter text”) did not hold for our tasks.

KV pool 732,892 tokens, concurrency 1.40x at 512k. My architectural prediction was 1.37x, so the arithmetic holds. Both numbers describe the bf16 operating point YaRN was measured at, not today’s state: with fp8 KV the same pool sits at 1,519,365 tokens and 2.90x. That does not make the YaRN arithmetic wrong, it just makes it per operating point. Throughput at 512k: 600.2 tok/s at c=8, +13 % against the 262k setting, TTFT 84 ms (262k: 90 ms). The first request after pod start took 39 seconds, and 3.78 in the second run: cold start, not a YaRN effect. Totals over individual measurements hide exactly this. Without the breakdown I would have credited YaRN with a slowdown it never caused.

MTP: there is no single right depth

The drafter is on, and depth is a lever that depends on the prompt. Measured on the same test instance, depth 4 against 6 against 8:

Prose wants 4, code wants 6, structured output wants 8. Acceptance rate falls from 0.505 to 0.433 while average length climbs from 2.02 to 3.46. Watch the length. And: reasoning_effort=xhigh (the model default) costs 28 % of acceptance. Anyone calling without effort=low gives up real speed.

Turning MTP off entirely is slower in every case (short prompts −21 %, 97k −11 %), even though a forum thread advised the opposite. Your own measurement beats the thread.

What landed in production, and what I threw away

Two levers are in, each proven on its own, because two levers flipped together produce a difference you cannot attribute:

The single request stays untouched by both, and the proof that the bottleneck was real rather than calculated: after the change the log said Running: 32 reqs, Waiting: 21. With the old 8 slots, 45 requests would have been queued.

The more interesting list is the one of things I tried and threw out again. It is longer than the list of winners, and every entry has evidence from the source code or the startup log, not a guess:

Two of those are the actual lesson. mamba-cache-mode all and the FlashInfer prefill backend are silently ignored: vLLM accepts the flag, logs the fallback at startup and carries on. You see nothing in the throughput, because nothing changed. Compare only the numbers and you have measured two identical configurations against each other, and then you call the result spread.

The prefill does not hang on the chunk budget

--max-num-batched-tokens sat at the default, and the default is 2048. That is not in the docs, it is in the source: DEFAULT_MAX_NUM_BATCHED_TOKENS in vllm/config/scheduler.py. vLLM itself warns at startup that you should raise it.

My hypothesis was that 2048-token chunks are too small for efficient matrix multiplication, so the single-request prefill has to get faster with a bigger budget. Written down beforehand so the measurement could refute it: above 3,200 tok/s on a single request.

Measured, aggregate prefill in tok/s:

The hypothesis is dead: 2,872 instead of the required 3,200, the single request gains nothing but spread. The gain sits in the shape of the curve. With 2048 the throughput dropped under load, from 2,782 to 2,543. With 8192 it stays flat. That is the behaviour of a shared budget: at 2048 with 32 requests in prefill each one got roughly 64 tokens per scheduler pass. So it is a scaling gain, not a peak gain, and the wall at about 2,900 tok/s does not move at four times the budget.

The price is the part where I was wrong. I had assumed the activation peak grows linearly and concluded the context had to come down from 512k to 262k. The startup log says: peak 2.9 GiB at 2048, 2.9 GiB at 8192, unchanged. The KV pool shrinks by 16,715 tokens, that is all. The planned context reduction died before it was rolled out.

The half I still cannot explain

And then there is the number I could not resolve. 46.65 ms per decode pass measured, 10.57 ms computed from weights and bandwidth. A factor of 4.4, and it is independent of context length, which is what makes it properly annoying.

What I ruled out, each with a measurement rather than an argument: PCIe (55.5 GB/s host to GPU is 88 % of Gen5, so the link is not it), kernel launch overhead (CUDA graphs are active, shown in the startup log), the MTP head (12.7 % of step time, which does not cover a factor of 4.4). That leaves the 48 GDN layers running on the Triton reference kernel as the most likely candidate, because the fused kernel requires SM90 or SM10.x. Newer builds have it but disable it with MTP active because of the 48:16 head ratio, where 8:1 is required. Upstream open.

This is the wall where this tuning arc ends, and it does not move with flags. Everything up to here was configuration decisions. What is missing here is a profiler run and probably a kernel I am not going to write. I am putting it in because a post that only shows the solved parts gives a false picture of the work.

Four measurement traps, and all four fired

1. Foreign load. 279 tok/s on the busy chat node against 984 on the empty batch node, identical config. A require-idle check that aborts instead of measuring.
2. CUDA graph capture. The first batch at a new width takes 40 to 85 seconds and delivers 6.5 tok/s. That is not a measurement. Warmup gets discarded.
3. Spread. Standard deviation 13 to 17 on a mean of 145. My “A beats B” was never a finding with that spread. Now it is a median over four repetitions.
4. Silently ignored flags. mamba-cache-mode all falls back to align, gdn-prefill-backend flashinfer falls back to Triton. Both visible only in the start log, never in throughput. An A/B built on them compares two identical configs.

And a fifth one that hit the measuring tool itself: the depth sweep reported “flag not found” because vLLM logs the flag inside quotes. The depth did take effect. A control check belongs on the level where the effect lives, not on the level of log text.

Along the way, because it took three weeks to believe: nvidia-smi, lspci and sysfs report a constant PCIe Gen1 at 2.5 on these boards, even under full load. That is wrong, a reporting artifact on AMD root complexes with ASPM. Only measured host-to-GPU throughput is evidence.

One name, two instances

Chat and batch jobs sit behind a single model name. Two identical instances, one on each GPU node, and the proxy distributes by least-busy. Measured: 16 parallel requests, 8 on one node, 8 on the other, zero errors.

Why this was worth writing down at all: of 127 cron jobs, 124 ask for the model name and pick no instance. Both instances looked identically loaded from outside, but cache hits stood at 125 million on one and 12.6 million on the other. One name covering both turns that skew into a routing problem instead of an architecture question.

Addressing a specific instance is no longer possible either, and that is deliberate: the older per-instance names all resolve through aliases onto the same group. Writing the old batch name lands you back in least-busy through the alias. It takes away the option of sorting the cache by hand, which is precisely why the routing has to do it.

Failover is the cheaper half: two retries and a 60 second cooldown inside the group. If one instance dies, the other carries it. The timeout chain has to grow monotonically outward, otherwise the top layer aborts before the fallback even starts. The real trap: the active timeout is not necessarily the one you set. The route said 900 seconds, the abort came at exactly 300.0: a cluster-wide idle timeout on the gateway that sits behind every service. An abort on a round number is always a config, never the model.

Session stickiness is the half where I changed my mind twice. The logic sounds obvious: keep a session on its instance so the prefix cache stays warm. Measured price: the same context on the other instance is completely cold. First token goes from 1.23 seconds to 4.65, hit rate from 93 % to 0, factor 4.8. A load balancer that switches between two instances pays that price on every second request.

For a long time my notes said the proxy could not do this. That was a statement about the version I was running, not about the tool. The current one can, and since August 28 it does: one shared Redis behind both proxy replicas, one session ID per conversation, TTL 600 seconds. Accepted the same day: six requests with the same session ID, the same instance six times, and the pin lived in Redis rather than only in the memory of one replica. That is the part that matters, because only then does it also hold for the second proxy.

What is still open is the caller contract: clients have to send the session ID with the request, and mine do not yet. Until they do, least-busy keeps running and the 4.8-factor cache handover is the price. Whether the pin actually carries it is the next A/B, not the next argument.

The proxy itself runs highly available: two replicas on two different nodes with anti-affinity, a PodDisruptionBudget that allows one disruption, Postgres behind it also doubled.

Uncensored, without the personal parts

Same 27B, a third checkpoint: the de-aligned NVFP4 variant, its own slot on the chat node, because both reach for the same four cards and cannot run side by side. The invocation, as it stands in the cluster:

exec vllm serve /model/models/dense-27b-uncensored-nvfp4 \
  --served-model-name=dense-27b-uncens \
  --host=0.0.0.0 --port=8500 \
  --tensor-parallel-size=4 \
  --trust-remote-code \
  --gpu-memory-utilization=0.85 \
  --max-model-len=524288 --max-num-seqs=8 \
  --max-num-batched-tokens=8192 \
  --attention-backend=TRITON_ATTN \
  --kv-cache-dtype=fp8_e4m3 \
  --mamba-ssm-cache-dtype float16 \
  --enable-chunked-prefill --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice --tool-call-parser=<xml-parser> \
  --reasoning-parser=<reasoning-parser> \
  --speculative-config='{"method":"<mtp-method>","num_speculative_tokens":4,"attention_backend":"TRITON_ATTN"}'

This instance has since moved onto the same operating point as the standard one: 512k, max-num-seqs 8, fp8 KV on TRITON_ATTN, MTP depth 4. The one remaining difference is the one that counts: the --served-model-name, with no alias. That is not an equivalence trick. The name dense-27b on the standard endpoint does not resolve here. Anyone asking for it gets the standardised variant.

The safety property is the part I want to name first. This model is not on the standard endpoint. It has its own served model name, with no alias, its own service and its own selector. Whoever asks for the chat model name gets the standardised variant, full stop. Crons, distillers, watchdogs and public traffic cannot land on it by accident, and there is no switch that redirects them. The uncensored model is a separate endpoint with its own access path, not a position on a dial.

Three traps from the model card, all three measured upstream and adopted here:

1. reasoning_effort has to be medium. That is not tuning, it is the operating condition. At the default the model burns its whole token budget inside the thinking block: it grows past 18,000 characters, sometimes degenerates into a loop, and no answer arrives. Upstream backs this with a nine-case gate in three configurations, 27 runs. At medium it is 1,000 to 3,000 characters and every run delivers.
2. The 15 MTP modules are in the quantization exception list and have to stay there. If the engine treats the bf16 MTP head as NVFP4, the drafter fails quietly: 0 % acceptance, slower than having no drafter at all.
3. W4A4 is the only path. The engine rejects W4A16 because the tile size does not work out. Set no quant flag, the engine reads it from the config.

In exchange, this checkpoint is the leanest of the three compared: 4.79 GiB of weights per rank against 5.45 on the standard. The KV pool follows the same fp8 decision as the standard endpoint. The model card also serves fp8, and with TRITON_ATTN on both the main and the draft layer that is the stable path here too. New checkpoints otherwise start at their declared context length; here the same YaRN extrapolation runs as on the standard, because both instances form one model group and need one shared window.

The 125B on two 72 GB cards

The second model in this post is a different weight class: a 125B MoE with 512 experts per layer plus a 51B ngram table (PLE), together 126 GiB across 419 files. The two RTX PRO 5000 72GB cards in this cluster sit in two different hosts. Node-local TP2 is physically impossible, so this is cross-node TP2 over Ray and RoCE, exactly the frame the 27B instances have been running on for weeks. The invocation on the leader; the worker side is identical except for the Ray address:

exec vllm serve /model/models/flash-next-uncens-nvfp4 \
  --served-model-name flash-next \
  --host 0.0.0.0 --port 8502 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --trust-remote-code \
  --load-format safetensors --safetensors-load-strategy lazy \
  --gpu-memory-utilization 0.98 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --no-enable-flashinfer-autotune \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --mamba-ssm-cache-dtype float16 \
  --enable-chunked-prefill --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice --tool-call-parser <xml-parser> \
  --reasoning-parser <reasoning-parser>

Three values here are the actual story:

• --gpu-memory-utilization 0.98. Weights 125.96 GiB against 131.9 addressable. At 0.95 you keep 2.3 GiB of reserve, too little for the NVFP4 swizzling overhead. 0.99 buys no usable KV reserve, disproved with a byte-for-byte identical failure signature. The arithmetic lives in a script in the repo, not in my head.
• PLE_FORCE_FP8=1 in the env, not a flag: the one-line patch that loads the 51B ngram table as FP8 instead of bfloat16. Without it 95.37 GiB, with it 47.68. The intermediate step was selective CPU offload (--cpu-offload-gb 13), which only moved the OOM from VRAM into node RAM. The patch replaces it, and the offload is out of the line again.
• --no-enable-flashinfer-autotune plus --compilation-config FULL_DECODE_ONLY. Both come from the official recipe for this RTX variant; the autotune would degrade the decode path on these cards.

Staging: 126 GiB from a Hugging Face host. The first run hung after 27 GiB, the second got throttled to 0.1 MB/s after an abort, while the mirror served the same files at 24.1 MB/s in the same window. Factor 240. The standard library does not handle the redirect plus Xet storage, and the recommended transfer accelerator is the wrong one for Xet. So: my own script with range resume, verified locally, 21.9 MB/s, and a deliberately truncated file refilled correctly.

The OOM that was not a context problem. The first start at full 262k context died on VRAM. The obvious theory (context too large) fell on an A/B: 32k and 262k produced a byte-for-byte identical OOM, 70.95 against 71.12 GiB, in 400 MiB allocation steps. Identical after changing a parameter clears that parameter. The cause sat in the weight load: the engine built the ngram table as bfloat16, so 95.37 GiB instead of 47.68 (FP8), because the checkpoint’s quantization config did not match the expected type and an isinstance check passed through it. The correct FP8 class existed in the image and was never picked. The fix is one line, gated on an env, placed before the type check. That 51B table is also why this model is a cross-node model at all: as BF16 it needs more than one card per rank on its own.

Offload was the road that did not work. The official recipe offloads the table into host RAM. Cross-node that is structurally blocked: the offload hook exists only in the single-process executor, and the config check requires nnodes == 1. Plus my own rule: do not take host RAM, everything goes on the GPUs. The generic selective offload (--cpu-offload-gb 13 --cpu-offload-params ngram_embedding) still carries the VRAM arithmetic, and the number 13 has a measured reason: at 26 the error only moved somewhere else. Instead of a GPU OOM, the Ray monitor killed the worker, because it counts the whole node including the 27B instance with its 23 GiB, and it trips at 95 %. With 13 GiB offloaded, 49.7 GiB stay on the GPU and the node sits at 72 %.

GPU pinning by UUID. Despite a DRA claim the container sees all five cards on the node, and the nvidia-smi index of the 72 GB card is 1 while the torch index is 0. The order flips between the two views. An index pin lands quietly on the wrong card. CUDA_VISIBLE_DEVICES with the UUID hides the other four and the rank sees exactly one card.

Probes, because startup takes 270 seconds. Without a startupProbe the container counts as started right away, and the group restart policy rebuilds the entire two-node group the moment the pod flinches during load. Measured: pod replaced every three minutes, the load time never finished. So a startupProbe with 20 s × 60 = 1200 s budget, longer than startup, otherwise the restart policy spins forever. The readinessProbe went the other way: 60 seconds of budget against 270 seconds of load, and under group rebuild a single not-ready is enough to rebuild the whole group endlessly.

The MTP against context trade-off. The MTP head costs 2.24 GiB (model loading 64.36 instead of 62.12). What is left is 2.07 GiB for the KV cache, and 262k tokens need 5.68. Maximum context with MTP: around 95,500 tokens. Measured, n=12 per variant, same prompts and seeds:

So MTP brings +52 % but does not combine with 262k on this checkpoint. An older note (“MTP costs 0 GiB, worth +2.9 %”) was measured wrong: the number came from a pod sitting in a restart loop and belongs to a different run. Only recognisable by the process ID.

That it hangs on this checkpoint and not on the model is the actual point of the balance sheet. 3.61 GiB are missing, so a checkpoint would have to be about 6 % smaller without losing quality. That sounds like a search task, and that is what I did next.

Screening 123 candidates without downloading 126 GiB

Since 262k and MTP do not fit together on this checkpoint, the obvious idea was a different checkpoint, with a smaller MTP head or a better packed PLE. So, go looking.

The run-up had already shown how expensive guessing is. One candidate was fully downloaded, 102 GiB, and then failed at the loader with ValueError: There is no module or parameter named 'ngram_embedding.shard_0'. An afternoon for information that was sitting in the file header.

A safetensors header lives at the start of the file, and HTTP can read ranges. So a range read of a few kilobytes per shard is enough for tensor names, dtypes and shapes:

u = f"https://huggingface.co/{repo}/resolve/main/{shard}"
n = int.from_bytes(get(u, rng="0-7"), "little")   # header length
hdr = json.loads(get(u, rng=f"8-{8+n-1}"))        # names, dtype, shape

That answers, before any download: size per dtype group, is there an MTP head, is the PLE in FP8 or BF16, how many experts per layer. The filter chain across 123 repos:

The exclusion reasons are more interesting than the result. MXFP4 looks like a fit on paper, the scheme gate reports get_min_capability() = 80, so “runs from Ampere onwards”. The gate that matters is a different one:

from vllm.model_executor.layers.quantization.utils.mxfp4_utils import mxfp4_experts_quant_supported
mxfp4_experts_quant_supported(120)   # False

The MoE kernels need sm100. On sm120 the whole thing falls back to BF16 dequant, so more VRAM and slower. Verified inside the container, not believed from the model card.

Another candidate carried our card class in its name and was 11 GiB smaller. The model card explains where from: 512 down to 448 experts per layer, 64 experts removed from each of the 48 MoE layers, calibration-free by mean|W| / RMS(W). That is 12.5 % less model, not a better quant, and not a single benchmark to go with it. Below that it said Requires a patched SGLang, stock builds emit GARBLED OUTPUT.

Two repos, one checkpoint

For the uncensored variant there were two candidates from the same author, one as UNCENSORED and one as ABLITERATED. The question was which one is better. The question has no answer:

207 files in both repos
identical sha256 : 207
different        :   0

The same checkpoint, published twice. Checked through the blob hashes in the API, without downloading. An A/B across two identical file trees would have produced noise that looks like a result, and I would have interpreted it.

A formula that was 23 GiB off

For the size estimate I needed VRAM per GPU from the on-disk size. My first formula: weights without PLE, divided by two cards. That produced values for every candidate that were 23 GiB off.

The only reason I noticed is that I checked it against a case with a known answer. The running checkpoint reports 62.12 GiB per GPU, and the 206 shards on disk can be recomputed:

TOTAL 125.87 GiB    MTP 4.86 | PLE 47.75 | MoE 63.72 | rest 9.55

Formula                 Result     Delta to 62.12
total/2                    62.94      0.82   ← hits
(MoE+rest)/2 + PLE/2       60.51      1.61
total − PLE                78.13     16.01
(total − PLE)/2            39.06     23.06   ← my first formula

Everything is sharded across both cards, the PLE included. And the point where a foreign checkpoint looked better was not a difference at all: ours has had the PLE in FP8 all along. Without that calibration I would have recommended a 115 GiB download on the basis of an assumption I never checked.

The download that failed twice

126 GiB from a Hugging Face host, and the standard library was the wrong way in. snapshot_download hung at 27 of 126 GiB, process alive, TCP connection in CLOSE_WAIT. After the abort the host throttled anonymous access to 0.1 MB/s, while a mirror served the same file in the same window at 24.1 MB/s. Factor 240. Point the library at the mirror and it dies on the HEAD redirect.

So, a script of my own, direct HTTP download with range resume against the mirror. Final state 126.0 of 126.0 GiB, verified afterwards: 206 of 206 shards, 0 broken headers, 296,475 tensors. And a deliberately truncated file was correctly topped up on the next run, so the resume is not just claimed.

The bitter part: both failures were already written as comments in the existing staging job in the repo. I reproduced them instead of reading first.

What sits at the end of the chain: the new checkpoint is structurally identical to the old one, same architecture, same 13 exclusion patterns in the quant config, 512 experts, PLE without packed tensors. That was precisely the condition for our FP8 patch and the 262k configuration to keep working unchanged. Per the model card the filter removal costs 0.18 percentage points of MMLU, measured over 2,280 questions with the same harness. Those are the author’s numbers, not mine, and that is how they stand here.

So the division of labour is settled: the 125B is the context setting, 262k native, no YaRN, no extrapolation, no MTP. The 27B stays the faster one with 205 tok/s single request, 984 aggregate and 512k through YaRN. Two different jobs, not replacements for each other.

The bug that gets to stay a bug

Last, the part where the right decision was to do nothing. With JSON-forced output the engine throws, reproducibly:

ERROR [backend_xgrammar.py:162] Failed to advance FSM for request
      chatcmpl-... for tokens 271. Please file an issue.

ERROR level, up to three waves per request. Token 271 decodes to '\n\n'.

Before chasing something like that, the question of whether anything is actually broken needs an answer. Ten consecutive requests with JSON mode and reasoning on: ten valid answers, zero broken, all with finish_reason=stop. The request recovers by itself, xgrammar just discards the draft proposal and lets the position be sampled again.

The trigger is cleanly measurable:

FSM errors before               : 3
after 3x JSON mode WITH reasoning : 6   (+3, so one per request)
after 3x JSON mode WITHOUT it     : 6   (+0)

My fix hypothesis was a config field called enable_in_reasoning, sitting at False. Name and context were a perfect match. The docstring says: “Whether to use structured input for reasoning.” That puts the grammar additionally on the thinking text, so more constraint, not less. Setting it to True would probably have made it worse. A config field whose name matches your own hypothesis is not evidence.

Upstream it is exactly the same case, open, with a different drafter: vllm#53777 describes the same token ID, the same grammar, the same retry waves, and calls it “a draft-vs-grammar interaction bug rather than corruption”. The related PR #44392 filters reasoning boundary tokens before the FSM advance, which also explains why it is '\n\n' specifically: whitespace at the transition out of the thinking phase lands in the accepted batch.

So it stays where it is. The obvious workaround costs reasoning, the other one costs 52 % speed, and the real fix belongs upstream. Building an alert for it would be the worst move: it would fire constantly in normal operation and hide exactly the case that matters, an FSM error with a broken answer.

What the 35B MoE could not do

The 35B MoE was the workhorse slot: chat, delegation, approval. Baseline on July 26, 2026: 155.7 tok/s single request, 806 at c=8, all on one node with short context. Two problems turned it into a problem.

First, stability. Above max-num-seqs 3 the model dies reproducibly from c=2 onward: HTTP 503, engine restart. Same symptom I had first attributed to batch width on the 27B. There the cause turned out to be the attention backend of the MTP draft layer, not the width. On the MoE I never ran that proof: it kept running on the old explanation, and max-num-seqs 3 ended any real parallelism. Whether TRITON_ATTN on both layers would also have lifted the threshold here is open. Retirement settled the question, but it belongs in this account honestly.

Second: cross-node was the weaker topology for this model. Those numbers are in the table below.

What the 27B then handled that the MoE could not:

Both numbers above come from the same tool on the same evening (August 12, 2026, 200 output tokens per request, c=8, both instances cross-node TP4). The instances were not configured identically: the 27B ran at 262,144 with max-num-seqs 8, the MoE at 524,288 with max-num-seqs 3. With 200 output tokens on short prompts the context sizes do not touch the measurement, but the MoE slot limit is part of its number: 356.7 tok/s is throughput under a stability cap, and that is precisely the finding.

And because this is the point where I caught myself: my retirement commit says “+45.7 % throughput at c=8 (600 vs 412 tok/s)”. The 600.2 is documented, that is the dense model’s 512k run. The 412 is not. I did not find it in any raw output, any measurement log, any results file: the value exists only in my own commit comment. It was back-computed from the percentage (600.2 / 1.457 = 411.9), and that percentage came from an earlier “+30 % ahead” against a MoE state I never measured alongside the 512k run. A number that exists only as the back-calculation of its own conclusion is not a measurement. It is replaced here by the pair that actually ran against each other.

The vision acceptance was the trigger, not just a nice number: a self-generated image with known content plus one random token that cannot possibly be in training material. Shapes, colours, OCR exact down to the digits, spatial relations, and the negative control: “cat in the image?” on an image without a cat, correctly denied. Without the negative control, four hits prove nothing.

The honest single number: sequentially the MoE is still faster, 140.7 against 108.0 tok/s in the same run as the table, because it moves 3B active parameters against 27B dense. What I am buying is parallelism, context and stability, not the single request.

The retirement, August 16, 2026: seven consumers repointed, and at first I had counted two. Full-text search across apps and scripts found seven, two of them NetworkPolicies that would have broken silently: the policies named the old port (the MoE listened on 8300, the 27B on 8500) and a label key the new pod does not carry at all. One policy had been ineffective for 13 days, traffic only passed because no matching ingress policy existed. Replicas 0 instead of deleted: manifest, PVC and Service stay, so whoever wants it back does not pay 17.5 GiB of staging again. The frontier model from the large US provider now runs only in emergencies, when local stalls.

What comes next

Two walls, both still open.

The first sits on the 125B: the MTP head. The current checkpoint lists it in the quant exception list, so it sits entirely in bfloat16 and costs 4.856 GiB of VRAM that the KV cache would otherwise use. There is an alternative community checkpoint in pure NVFP4 whose MTP head costs 1.475 GiB: with it, MTP and 262k fit on the two cards with reserve. Staged, untested, and that is where it stays until the long-context proof (needle in a haystack, 120,000 tokens) runs against the known state. You do not buy community quants, you measure them.

The second wall sits on the 27B: factor 4.4, 48 GDN layers on the Triton reference kernel, upstream still open. Once the fused kernel works on sm120 with MTP active (it wants an 8:1 relation, we have 48:16), the 27B gets a second boost, and the tuning numbers in this post become one operating point. The next A/B decides how much.

What I actually take away from those four weeks is none of the numbers above. It is the list of ten cases where I asked the wrong source:

None of those sources lied. Each answered correctly what I asked. The mistake was the same every time: I took the source that answers instead of the one that decides. And the answering one is always the more convenient, otherwise it would not be a pattern.

In practice that means the question before the measurement is not “what does X say”, it is “is X even responsible here”. It costs two minutes and in four weeks it would have saved me hours ten times over.