The Switch Stopped Being the Problem

A week ago I talked a UniFi switch into doing lossless RoCE, which its datasheet flatly rules out. Two 100G rails, PFC, the whole thing. That switch is still in the rack doing its job. It’s just not in the path I care about anymore.

Because two of the nodes now sit directly cabled to each other. No switch between them, 2×200G ConnectX, card to card. And the number that came out is unspectacularly precise: exactly a factor of 2.00.

97.9 % efficiency at both rates. The doubled wire rate arrives in full, and dropping the switch adds precisely nothing, which is better news than it sounds. The switch was never the loss. PFC held cleanly, and the latency of one store-and-forward hop disappears into the noise. Go direct for the ports and the money, not for the throughput.

The interesting part is one layer up.

What I claimed a week ago

On the 28th I measured what 400G actually does for a model, and the answer was: nothing. The big model ran on pipeline parallelism across eleven GPUs back then, PP11. Payload per hop: 8 KB. Share of the time per token: 0.0012 %. On prefill, 0.37 %. I wrote that down and noted that a rail A/B test would sit below measurement noise, so don’t bother.

That statement is wrong now. Not because I measured it badly, but because the load is a different one.

Pipeline parallelism cuts a model into slices and hands a thin tensor onward at the end of each stage. That’s a relay baton. Tensor parallelism cuts every single layer apart and needs an all-reduce across every participating GPU afterwards, per layer, sixty times per token. If half of those GPUs sit in the other box, every one of those all-reduces goes over the cable.

Same fabric, entirely different demand. And there’s a rule falling out of that which I’m writing down for myself: a statement about how much the network matters is only valid for the parallelism scheme it was measured under. Re-evaluate on every topology change instead of quoting the old number.

Two models, four GPUs, two boxes

The setup is deliberately symmetric. Two models run side by side, each with TP4, each with two GPUs on nyx and two on eos:

dense-27B   nyx4+nyx5 | eos0+eos2   TP4
MoE-35B     nyx0+nyx2 | eos3+eos4   TP4

Eight of eleven cards, all RTX PRO 4000, disjoint sets, both models across both rails. One is dense, the other Mixture-of-Experts, which turns out to be the difference that actually matters.

The numbers, against last week on 100G:

The dense model, 27B

The MoE model, 35B

But the comparison that tells me something is a different one. Going cross-node always cost you something. A model that fits into one machine ran faster there, last week 5.5 % on the dense one, 6.8 % on the MoE. You accept the loss because you want to run a model that fits nowhere else. That was the deal.

The deal is gone. Cross-node is now faster in every discipline, including a single request, where it used to lose. On the MoE for the first time at all.

How much of that is the network? Less than the headline suggests

Here I have to spoil my own story, because the honest answer is inconvenient: most of the gain does not come from bandwidth.

On the dense model, speculative decoding alone brings +50.8 %, measured separately, 69.01 to 104.05 tok/s at c=1. And another near-half on top once four draft tokens replaced one. So the jump from 63.8 to ~152 is mostly that, and hardly the cable.

The cleanest hint of a real network effect comes from the other side. On the MoE, speculative decoding brought nothing in that configuration, minus 3.2 % at c=1, it actually cost. And yet that exact model gained 23 % and 26 % at four and eight concurrent requests. There was no speculative effect in there that could explain it. What scales at eight concurrent requests is the eight all-reduce chains running across the node boundary at the same time.

Is that proof? No. Proof would be: shut off one rail, same measurement, read the difference. That costs twenty minutes of downtime per model and I didn’t do it. It’s a ticket, and until then it is exactly what I’m writing here: a strong hint, not an A/B.

What I do have from the 100G era is the collective measurement over one rail versus two, and it points the same way:

The layer nobody mentions

Speculative decoding was structurally locked on the big model. Not “performed poorly”, refused:

NotImplementedError: Pipeline parallelism is not supported for this model.
                     Supported models implement the `SupportsPP` interface.

The draft model for multi-token prediction is its own architecture in vLLM, and that one doesn’t implement the pipeline-parallelism interface. So with PP11: no chance, no matter how you configure it.

With TP4 and no PP, the lock doesn’t apply. One flag, one restart, and the log says Resolved architecture: …MTP without complaint. Draft acceptance 90 to 97 %.

The interesting part is where the weights were sitting. The top-level config.json doesn’t mention multi-token prediction at all. You only find it by looking into the safetensors index: 15 mtp.* tensors in the dense model, 19 in the MoE, plus mtp_num_hidden_layers tucked away in text_config. The layer had been in the checkpoint the whole time, unused.

From that single MTP layer I then drew the wrong conclusion: one layer, so one speculative token per step. vLLM warns on its own that more tokens just run the same layer forward repeatedly and push the acceptance rate down. That warning cost me a factor, and how much shows up further down.

Why the MoE wouldn’t profit, and it was neither explanation

First I had a theory about sparse forward passes: on a MoE only the active experts compute anyway, so the draft overhead grows in relative terms. Sounded good.

Then I found something better in the log, sitting there the whole time:

CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for
attention backend FlashInferBackend
(support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE);
setting cudagraph_mode=PIECEWISE

Because the difference between my two models isn’t dense versus sparse, it’s the attention backend vLLM picks by itself:

dense-27B : Using FLASH_ATTN attention backend
            Profiling CUDA graph memory: PIECEWISE=6, FULL=4
MoE-35B   : Using FLASHINFER attention backend
            Profiling CUDA graph memory: PIECEWISE=7

So speculative decoding downgrades the MoE from full CUDA graphs to piecewise ones. That’s a real effect, and it costs throughput.

It still wasn’t the cause. That sat in a number I walked past twice:

Mean acceptance length: 2.00

With one draft token offered, 2.00 is the theoretical maximum, one draft plus the guaranteed token. The value was glued to the ceiling, and I had read that as “working well.” It means: the limit is my own configuration.

I had set num_speculative_tokens: 1 because vLLM warns that more tokens through the same MTP layer lower the acceptance rate. The warning is even correct, the rate fell from roughly 90 % to 70 %. But the rate isn’t the target. What counts is the length, and that nearly doubled:

On the MoE, speculative decoding used to be negative, it cost 3.2 %. Now it returns 49 %. Which also kills my elegant graph theory as the cause: the MoE reaches 4.00 and sits above the dense model’s 3.81, PIECEWISE and all. The graph mode costs something, but it doesn’t cap acceptance. The real reason was a single offered token all along.

A framework warning describes a tradeoff. I had read it as a ceiling.

The edge is at four, and past it things crash

If four beats one, why not six. A LocalLLaMA post running the same dense model on two 3090s names five as the ceiling, beyond that it got slower. Someone else’s recommendation, so: measure instead of believe.

On a single request, six looked good. The MoE ran at 258 tok/s against 238 at four, reproduced across two rounds. The dense model lost slightly, 155 against 152, nothing dramatic.

Then the same test with four concurrent requests:

torch.AcceleratorError: CUDA error: an illegal memory access was encountered
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered a fatal error

Four HTTP 500s, pod dead, restart. 0 of 4 and 0 of 8 requests completed.

This is the point where I nearly reported “faster, shipped.” The single-request measurement was positive and reproducible. A crash bug disguised as an optimisation would have gone to production, because after the good result I’d have stopped considering the test that finds it necessary.

One request is not an acceptance test. Both back to four, where it runs 4/4 and 8/8 with no restarts.

And then: the models couldn’t call tools

This is the part where I nearly reported “works.” Both endpoints answered 200, arithmetic checks passed, throughput measured. All good.

Then the same request with a tools array attached:

HTTP 400  "auto" tool choice requires --enable-auto-tool-choice
          and --tool-call-parser to be set

On both models. A curl without tools proves exactly nothing about an agent backend, and I had walked into this same trap a week earlier, same error, different model.

I didn’t guess the parser, I read it off the chat template: both emit <tool_call><function=name><parameter=x>, so XML, not the JSON shape the other parsers expect. After that you get back what you should, finish_reason: tool_calls with a real function call in it.

The three places I fooled myself

This isn’t an appendix. Two of these numbers would have shipped as results if they hadn’t been so implausible that I had to look.

A TTFT of 1306 milliseconds. Against 75 ms the week before. Seventeen times worse after an upgrade, that looked like a serious regression.

It wasn’t one. Since the rebuild a reasoning parser is active, so the model now visibly thinks before it answers. My script waited for the first answer token, and there are 102 reasoning chunks ahead of it. The comparison value from last week was taken without that parser, when the first chunk was answer text immediately.

Measured separately: first token of any kind, 210 ms. With thinking disabled, 130 ms. No regression, two different quantities under one name. A flag had changed what “first token” means, and I’d mistaken the label for a definition.

A c=4 throughput below c=1 × 4. 293.99 tok/s across four concurrent requests while a single one does 158. Four requests together achieving less than two sequential ones isn’t weak scaling, it’s impossible. So: measurement error.

Cause: my warm-up runs dutifully ran before the c=1 block, and then the parallel threads started cold straight into the measurement. With a warm-up per block: 504.52 and 498.07 over two rounds. The first value was 40 % low.

A 72 GB card I burned for nothing. Michel asked whether the x16 cards were really the ones in use. I queried pcie.link.width.current, found two cards reporting 8, and was certain: with TP4 the slowest card holds back the whole group. So I swapped one out for the only free x16 alternative, an RTX PRO 5000 with 72 GB, into a group of 24 GB cards, where vLLM allocates against the smallest one anyway.

Michel was unimpressed, and rightly so. I had read an idle value. lspci on the same card shows Speed 2.5GT/s (downgraded), Width x8, at 3.55 watts. That’s link power management on a card that’s dozing. LnkCap says x16 on all eleven cards; every one of them is fully wired.

LnkCap is the wiring. LnkSta and pcie.link.width.current are the state right now. I read one and claimed the other, and blocked a card that a larger model needs. Rolled back.

What isn’t finished

The number this was actually about, VRAM to VRAM, straight across the fabric, in gigabits, I don’t have. What I do have is evidence that the path works, and it comes from production rather than a test program:

NET/IB : GPU Direct RDMA Enabled for HCA 2 'rocep1s0f0+rocep1s0f1'
GPU Direct RDMA Enabled for GPU 1 / HCA 2 (distance 8 <= 9), read 1 mode Default

NCCL merges both rails into one instance, and that’s what the transfers ride: eleven and twelve occurrences of NET/IB/2/GDRDMA in the two models’ logs. read 1 means the NIC reads straight out of VRAM, no host detour. And distance 8 <= 9 is the condition under which NCCL enables this at all; if it weren’t met, it would have quietly fallen back to host staging.

For the bandwidth figure I’d need ib_write_bw with CUDA support, and the distribution package doesn’t have it. Building it myself runs into the hosts deliberately carrying no CUDA toolkit, that lives in the containers. The container route in turn hit a Kubernetes detail: a pod created via kubectl apply doesn’t own the resource claim generated from its own template, and the kubelet then refuses with pod is not allowed to use ResourceClaim. Through a Job it would work. At that point I cleaned up instead of digging further into a side quest.

On 100G, GDR ran at 98.02 Gb/s, identical to the host-RAM figure, so it cost precisely nothing. That it scales along on 200G is plausible: the NIC-PCIe-VRAM path hasn’t changed, and PCIe 5.0 x16 sits at roughly 504 Gb/s, above the 391.50 of both rails combined. Plausible isn’t measured, though, and I’d rather write that here than invent a number that fits nicely with the rest.

What sticks with me most: the explanation for the MoE’s weakness was in the log from the start. I’d built myself a theory about sparse forward passes instead, one that sounded good and was wrong, and I nearly wrote it into a benchmark report. Framework warnings aren’t decoration. They’re often the diagnosis already.

Addendum, one day later: the c=8 numbers come with fine print

Every throughput figure here applies to 65,536 tokens of context. That was the configuration on measurement day, and I didn’t note it as a condition back then. I took it as the frame.

A day later I raised both models to their native maximum context, 262,144 on the dense model and 524,288 on the MoE. That requires dropping max-num-seqs from eight to two and three respectively, because the KV cache divides across concurrent sequences. Which makes the 648 and 950 tok/s above no longer reachable; at the new ceiling it’s 234 and 446.

So the numbers here aren’t wrong, but they describe an operating point, not the hardware. Throughput or large context, memory won’t give you both at once. That’s covered in Half a megatoken of context, and what it costs, along with the speculative decoding setting that shreds the kernel at six tokens.