Half a megatoken of context, and what it costs
Last week two nodes sat directly cabled to each other, 391 Gb/s across two rails, and two models ran faster cross-node than on a single machine. Bandwidth was solved. What I hadn’t touched: both models were running with 65,536 tokens of context while they were capable of 262,144 and 524,288 respectively.
A quarter and an eighth of what was available. That’s fixed now, and getting there produced two things I didn’t know before.
The numbers first
| before | now | actually measured | |
|---|---|---|---|
| dense-27B | 65,536 | 262,144 | 261,899 tokens, needle found |
| MoE-35B | 65,536 | 524,288 | 521,315 tokens, needle found |
The right-hand column is the point. A configuration claims a context length, it doesn’t prove one. I put a memorable sentence at the beginning of a prompt, the hardest case, because silent truncation drops exactly that first, and then asked for it. Both models quoted it back, at 99.9% and 99.4% of their respective ceilings.
Above the limit the API refuses cleanly (maximum context length is 524288), 0.2
seconds, no crash, no silent truncation. A hard, correct edge.
Both models are natively trained to these lengths, no RoPE scaling involved. That’s why I did not push the dense model to 512k even though it would technically work: a factor of two above the training length trades retrieval quality for a prettier number in the manifest. The prompt gets accepted, but what the model still finds in the middle is unverified and usually worse.
The price shows up in one number
Context doesn’t cost speed. It costs parallelism.
The KV cache is the hard limit, and it gets divided by the number of concurrent sequences. At 65k both models ran eight requests in parallel. For full context I had to come down:
max-num-seqs before | now | |
|---|---|---|
| dense | 8 | 2 |
| MoE | 8 | 3 |
Which means the throughput numbers I wrote up last week as a result are no longer reachable:
| at 65k | at full context | |
|---|---|---|
| dense, 8 parallel | 648 tok/s | no longer possible |
| dense, at the limit | 234 tok/s (2 parallel) | |
| MoE, 8 parallel | 950 tok/s | no longer possible |
| MoE, at the limit | 446 tok/s (3 parallel) |
Single-request throughput is untouched: 163 and 238 tok/s, identical to before. Anyone running a model for an agent with a huge context loses nothing. Anyone serving many parallel requests loses half their throughput.
That isn’t an optimization, it’s a decision. And I hadn’t been treating it as a decision, I thought “more context” was a feature you switch on.
A surprise in the good direction
My prediction was 2.23x concurrency headroom for the dense model and 3.30x for the MoE. Actual:
dense: 707,017 tokens KV cache -> 2.70x for 262,144
MoE: 2,222,981 tokens KV cache -> 4.24x for 524,288
The KV cache grew at unchanged gpu-memory-utilization. Fewer concurrent
sequences need less CUDA graph memory, and what’s freed lands in the cache. My
arithmetic was too conservative, a pleasant error, but an error.
The ratio is what’s remarkable: at practically the same memory footprint, the MoE holds three times as many tokens in cache as the dense model. MoE layers need less KV per token. That’s why half a megatoken is nearly free on the MoE while the dense model already has to drop to two sequences for 256k.
The speculative decoding that only dies under load
Second front, same week. Both models run multi-token prediction, and the parameter for it was set to 1, a single drafted token.
On the MoE, MTP was actually negative: 3.2% slower than without it. I had a theory about sparse forward passes and relative draft overhead. The theory was wrong. It was just the parameter.
| MoE | 1 token | 4 tokens | |
|---|---|---|---|
| acceptance length | 1.85 | 4.00 | +116% |
| single | 154 tok/s | 238 tok/s | 1.54x |
| 8 parallel | 867 tok/s | 951 tok/s | 1.10x |
An acceptance length of 4.00 with four tokens offered is the theoretical maximum. Fully saturated.
So the obvious next step: six tokens. On a single request it was better, 258 instead of 238 tok/s, reproduced across two runs. I was one sentence away from reporting that as an improvement.
Then four requests in parallel:
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
vllm.v1.engine.exceptions.EngineDeadError
Zero of four. And zero of eight. Pod dead, restart, model gone.
This is the kind of bug that ships to production disguised as an optimization. On a single request it isn’t merely invisible, it looks good. One request is not an acceptance test, and I would have considered the concurrency check unnecessary if the first number hadn’t been so tempting.
Both back to four. There it’s 4/4 and 8/8, zero restarts.
A number I have to retract
On the dense model I measured 161.91 tok/s after the MTP change. That number does not reproduce. After a rollback and three repeats: consistently 151 to 153, at identical configuration.
Four hypotheses tested, all refuted. Thermals sat at 39 to 62 °C, full clocks. Different GPUs from a scheduler reassignment: the same eight. Degraded CUDA graphs: that turned out to be a truncated grep in my own check, full graph mode had been there all along. The second model stealing resources: MoE scaled to zero, dense measured alone, 152.09, no difference.
Against a genuine regression stands the acceptance length. 3.97 today versus 3.81 back then, so higher. The 161.91 was an outlier I had written down as a result. The defensible figure is ~152.
That’s uncomfortable to write, but a number you’ve seen once and never reached again isn’t a measurement. It’s a coincidence with decimal places.
What I didn’t measure
Two gaps I’m naming explicitly, because otherwise they read like results.
Retrieval in the middle. The needle sits at the start of the prompt. That’s the hardest single case, but a proper needle-in-a-haystack probes multiple depths. At 500k, “finds the beginning” does not mean “finds everything”.
The prefill timings are useless. The 500k run was faster than the 400k run, 41.8 versus 83.3 seconds. That’s prefix caching, the test prompts overlap. Fine as a functional proof, worthless as a performance measurement.
The actual takeaway
Going from 65k to half a megatoken was technically unremarkable: two numbers in a manifest, one rollout, six minutes.
What was interesting is what fell out along the way. A speculative decoding setting that only reveals it’s broken under load. A KV cache that grows when you give it fewer sequences. One of my own measurements that didn’t survive a repeat. And a tradeoff I had mistaken for a feature.
Three of those four came from repeating a measurement that had already produced a result. That’s the most boring part of the work and apparently the most productive.