Unlocking lossless RoCE on a UniFi enterprise switch

If you have read the earlier posts, you already know the shape of me: I run a GPU cluster down in the basement, and I would rather spend a weekend elbow-deep in it than reach for a credit card. You can hear the fans spin up through the floor when something starts working hard. This is the story of one of those weekends, the one where I taught a switch to do something its datasheet swears it can’t.

The cluster has been doing inference for months, quietly, each model sitting on its own card. That worked until the first model arrived that no longer fit on a single GPU. Fitting a bigger model on one box only gets you so far. Tensor parallelism inside a single machine, TP=8 across the GPUs in that chassis, is straightforward: they share NVLink or PCIe, the interconnect is fast and local. But once you want to run something genuinely large, and there is spare VRAM sitting idle in the other hosts, you have to go wider than one box. TP=16 across two machines, or more. Once tensor parallelism crosses the host boundary, the network stops being plumbing and becomes the thing that decides whether it works at all.

So the problem is splitting a model across nodes without the network becoming the bottleneck. Tensor parallelism is the demanding case, because every layer triggers an all-reduce across all participating GPUs. Within a box NVLink swallows that. Across hosts it lands on the Ethernet, and ordinary Ethernet is nowhere near good enough: the GPUs have to talk constantly, without the CPU in the way. The tool for that is RDMA, and over Ethernet that means RoCEv2, and RoCEv2 needs one thing the rest of your network never asks for: a lossless fabric. In practice, a switch that does Priority Flow Control. The textbook answer is to go buy one. A used Mellanox/NVIDIA Spectrum SN2700 runs 2,500 to 3,700€ on the German market right now, and the cheap US listings land in the same range once you add 19% VAT, customs, and shipping. New is comfortably five figures. Or you fight a MikroTik into submission.

I did not want to buy another switch. There was already one sitting in the rack, humming along: a UniFi aggregation switch. Everything I read said it couldn’t do RoCE, no PFC, no DCB, nothing. So the obvious question, the one nobody seemed to be asking, was simply: why not? What is actually stopping it?

So I SSH’d into it. The UniFi Enterprise Campus Aggregation switch runs SONiC under the hood. Below the locked-down web UI is the same open network OS the hyperscalers use, on a Marvell ASIC that is fully capable of Priority Flow Control. The capability was there the whole time. Ubiquiti just doesn’t expose it.

From there on it stopped being a purchasing problem and became an understanding problem. And before I show you how I solved it, and how I took the switch completely offline in the process, a quick word on what all this effort is even for. Because that decides whether any of it matters to you.

The workload, and why the network is the wall

The goal is a dedicated inference and training fabric for a multi-node GPU cluster: Kubernetes-native, NVIDIA GPUs, serving and training large models that do not fit on a single box. When a model spans nodes, the GPUs have to talk to each other constantly. All-reduce, all-gather, KV-cache movement. That cross-node traffic is the bottleneck, and the way you make it fast is RDMA: let the network card write straight into GPU memory (GPUDirect RDMA), bypassing the CPU entirely.

RDMA over Ethernet is RoCEv2. And RoCEv2 has one hard requirement. The network must be lossless. A single dropped packet tanks RDMA throughput, because the whole model assumes the fabric never drops. You get lossless Ethernet with Priority Flow Control (PFC): instead of dropping a packet when a buffer fills, the switch sends a PAUSE frame upstream and says “hold that priority for a moment.” That is the feature that turns ordinary Ethernet into an RDMA-capable fabric, and it is exactly the feature UniFi doesn’t expose.

RoCE in plain terms

If the last two paragraphs threw acronyms at you, here is the ground floor. Skip ahead if you already speak fluent RDMA.

Normally, when one machine sends data to another, the CPU is in the middle of every step: the network card hands the packet to the operating system, the OS copies it into application memory, and only then does your program see it. Fine for a web request. A disaster when eight GPUs need to exchange gigabytes of tensor data thousands of times a second, because the CPU and those memory copies become the bottleneck.

RDMA (Remote Direct Memory Access) removes the middleman. The network card on one machine writes directly into the memory of the other machine, without either CPU getting involved. GPUDirect RDMA goes one step further: the card writes straight into GPU memory. The data lands where it is needed, nobody copies it twice, and the CPUs stay out of the way. That is what makes cross-node GPU work fast enough to be worth doing.

RoCE (RDMA over Converged Ethernet, pronounced “rocky”) is how you run RDMA over an ordinary Ethernet network instead of expensive InfiniBand gear. Version 2, RoCEv2, is the one everyone uses today. It gives you the direct-memory magic on the same kind of switches and cables you already run.

There is one catch, and the whole rest of this post hangs on it. RDMA assumes the network never drops a packet. Regular Ethernet does not promise that: when a switch gets more traffic than it can handle, it simply throws packets away and lets the sender notice and retransmit. TCP is built to absorb that. RDMA is not. A single dropped packet doesn’t just cost that packet, it stalls the whole transfer and throughput falls off a cliff.

So the network has to be lossless, and that is what Priority Flow Control (PFC) buys you. Instead of dropping a packet when a buffer fills up, the switch sends a small PAUSE signal back to the sender: “hold this class of traffic for a moment, I need to catch up.” The sender waits a beat, the buffer drains, traffic resumes, nothing is lost. PFC is the single feature that turns ordinary Ethernet into a fabric RDMA can trust. And it is exactly the feature UniFi ships in the silicon but doesn’t expose.

Two things worth stating plainly before we go under the hood:

• Yes, this carries training too, not just inference. Meta runs distributed AI training at scale over RoCE. The requirement is identical, lossless fabric plus PFC plus ECN, whether you are serving a big model across nodes or training one. Inference is the gentler workload. If the fabric holds for training’s all-reduce storms, inference is easy.
• This is not NVIDIA-only, with one honest caveat. The switch work here is completely vendor-neutral: it makes the fabric lossless, and any RoCEv2-capable NIC benefits. The Intel E810 (irdma driver) runs RoCEv2 just fine, and the new E835 (up to 200GbE) lists it explicitly. But: doing RoCE is not the same as doing GPUDirect. The classic GPUDirect path (nvidia_peermem) only exists for Mellanox/ConnectX; the newer DMA-BUF path is vendor-neutral in design and irdma has the kernel support by now, but NVIDIA officially validates GPUDirect on ConnectX only, on an E810, RDMA transfers effectively land in host RAM and get copied to VRAM from there. For storage, CPU workloads and also for model serving the E810 is perfectly usable. If you want the last third of throughput on GPU-to-GPU traffic though, the card side has to be ConnectX.

What the switch actually is

UniFi switches feel like appliances. Web UI, a few toggles, no way in. So I SSH’d into it.

The device SSH credentials on UniFi are not the controller login. They live in the controller’s management settings and get pushed to the device. Once in:

$ show version
SONiC Software Version: SONiC.3.0.8
Distribution: Debian 10
Kernel: 5.10.0
ASIC: marvell
HwSKU: ECSAGG-48x25G6x100G
Platform: x86_64-marvell_db98cx8514_10cc-r0

That is not a UniFi firmware. That is SONiC, the open-source network OS from the Open Compute Project, the same NOS that runs in Microsoft Azure and a good chunk of the world’s hyperscale fabric, built by Ubiquiti on a Marvell Prestera db98cx8514 ASIC. The full toolset is right there: sonic-cli, vtysh, redis-cli, sonic-cfggen, config, show. The config lives in a redis database exactly like any other SONiC box.

Which means the hardware can, in principle, do everything SONiC can do. That includes the full DCB stack: PFC, ECN, WRED, lossless buffer pools. The capability is in the silicon. The only question is whether Ubiquiti left the door open.

The mistake that crashed the switch

It had. And I still walked straight into a wall first.

SONiC has a one-shot command to load QoS and buffer config: config qos reload. Coming from a normal SONiC box, the instinct is obvious. Just reload QoS and let it generate the RoCE config. So I ran it.

It took down the entire switch. The orchagent, the process that programs the ASIC, crashed with a std::invalid_argument from stoul, threw SIGABRT, and took the whole swss container with it. The switch dropped off the network. Recovery was a reboot.

The chain that got me there is short and worth understanding, because it is a trap specific to these Ubiquiti-flavored boxes. Ubiquiti ships this hardware without the files a normal SONiC switch uses to size its lossless buffers (pg_profile_lookup.ini and the pg_lossless_* profiles are simply not there). So config qos reload trips immediately on a missing template. The obvious “fix” is to set DEVICE_METADATA.type so the template resolves, and that is the mistake: now the reload happily tries to generate buffer config for all 54 ports at once, except there is no cable-length table to compute the lossless headroom from. It emits malformed buffer keys, orchagent chokes on the first one it can’t parse, and the control plane goes down with it. It is a known SONiC failure mode (sonic-buildimage issue #16186), I just walked straight into it on a box that was primed for it.

The rules I learned the hard way:

• Never run config qos reload on this switch. It is global and it will crash a production box.
• Never touch DEVICE_METADATA.type on a live switch. It rewrites the entire buffer model.
• config checkpoint and rollback do not work here. Ubiquiti’s custom config fields aren’t YANG-conformant, so the checkpoint validator refuses. Your only real fallback is config reload (reloads the on-disk config) or, worst case, a reboot.

If you do crash it: a plain reboot brings it straight back. The on-disk config_db.json is Ubiquiti’s clean baseline and does not contain your experiment, so the switch returns to its working production state. Nothing about the RoCE work below is persisted to disk unless you explicitly make it so (more on that later), which means a reboot is a guaranteed reset button. Back up first anyway:

sudo sonic-cfggen -d --print-data > /tmp/config_backup_$(date +%s).json

Why the feature isn’t exposed, though the hardware has it

The buffer myth first. I initially assumed “24 MB of ASIC buffer is too small for RoCE.” Wrong. The MMU is 24 MB total, the ingress_lossless_pool is already carved out at ~20 MB, and the Marvell ASIC is explicitly on SONiC’s list of PFC-capable silicon. Buffer was never the blocker.

The real blocker is pure software packaging. UbiOS, the UniFi layer, does not expose PFC/DCB/QoS in its web UI at all. And the SONiC image ships without the lossless profile definitions that buffermgrd needs to generate PFC buffers. The chip can do it. The tooling is present. Ubiquiti just didn’t wire the last mile, and doesn’t surface it.

That is a product decision, not a hardware limit. It is a missed selling point, not a missing capability, and I come back to that at the end.

The architecture

Three pictures. First, where the switch sits. It straddles the ordinary cluster Ethernet and the dedicated RoCE fabric:

Second, the SONiC control path inside the switch. This is the chain your config travels, and the last hop is the only proof that matters:

Third, the actual RoCE data path with PFC doing its job. A PAUSE frame instead of a drop:

The fix: enabling RoCE port-by-port, safely

The safe path is the opposite of config qos reload. Touch one port at a time, write directly to the redis config_db, build the lossless buffer profile yourself, and check the switch’s health after every step. No global reload, ever.

A few things that will save you pain:

• Redis keys contain |. Quote them, or your shell will eat the pipe. Run everything from an uploaded script rather than nested SSH heredocs.
• References between config_db tables must use the [TABLE|key] format, or orchagent logs malformed reference ... must be surrounded by [ ] and silently skips them.
• Verify at the ASIC, not in config_db. The only thing that counts is the SAI attribute on the port.

The buffer profile values below are for 100G on a short DAC, derived conservatively from a known-good Centec SONiC reference. Adjust size, xon, xoff for longer cables.

Step 1, lossless buffer profile (the piece Ubiquiti left out):

redis-cli -n 4 hset 'BUFFER_PROFILE|pg_lossless_100000_5m_profile' \
  pool '[BUFFER_POOL|ingress_lossless_pool]' \
  size '36864' xon '18432' xoff '18432' dynamic_th '0'

Step 2, QoS maps (DSCP 24 maps to traffic class 3, the classic RoCE priority):

redis-cli -n 4 hset 'DSCP_TO_TC_MAP|AZURE'                 0 1 24 3 32 4
redis-cli -n 4 hset 'TC_TO_PRIORITY_GROUP_MAP|AZURE'       0 0 3 3 4 4
redis-cli -n 4 hset 'TC_TO_QUEUE_MAP|AZURE'                0 0 3 3 4 4
redis-cli -n 4 hset 'MAP_PFC_PRIORITY_TO_QUEUE|AZURE'      3 3 4 4
redis-cli -n 4 hset 'PFC_PRIORITY_TO_PRIORITY_GROUP_MAP|AZURE' 3 3 4 4

Step 3, bind the lossless PG and the QoS map to the port (here Ethernet48; note the [TABLE|key] references and pfc_enable):

redis-cli -n 4 hset 'BUFFER_PG|Ethernet48|3-4' \
  profile '[BUFFER_PROFILE|pg_lossless_100000_5m_profile]'

redis-cli -n 4 hset 'PORT_QOS_MAP|Ethernet48' \
  dscp_to_tc_map '[DSCP_TO_TC_MAP|AZURE]' \
  tc_to_queue_map '[TC_TO_QUEUE_MAP|AZURE]' \
  tc_to_pg_map '[TC_TO_PRIORITY_GROUP_MAP|AZURE]' \
  pfc_to_queue_map '[MAP_PFC_PRIORITY_TO_QUEUE|AZURE]' \
  pfc_to_pg_map '[PFC_PRIORITY_TO_PRIORITY_GROUP_MAP|AZURE]' \
  pfc_enable '3,4'

After each step, confirm the control plane is still alive before continuing:

docker ps --filter name=swss --format '{{.Status}}'   # must say "Up ..."

The proof

config_db is a wishlist. The ASIC is the truth. Pull the SAI attribute for the port and check that PFC is actually programmed into silicon:

OID=$(redis-cli -n 2 hget 'COUNTERS_PORT_NAME_MAP' 'Ethernet48')
redis-cli -n 1 hget "ASIC_STATE:SAI_OBJECT_TYPE_PORT:$OID" \
  SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL
# returns 24    (binary 11000 = priorities 3 and 4 enabled)

24 means priorities 3 and 4 have PFC enabled at the hardware level. The switch now sends PAUSE frames instead of dropping on those priorities. That is a lossless fabric, on hardware whose spec sheet doesn’t list the feature.

Then the end-to-end test with real RDMA traffic across the switch:

# server on one port, client on the other, RDMA write over the fabric
ib_write_bw -d <hca> -F --report_gbits -R            # server
ib_write_bw -d <hca> -F --report_gbits -R <server>   # client

Update, a few days later: the full numbers are in. Three nodes with dual-port ConnectX-5 Ex hang off the fabric now, all firmware levelled, and I measured the complete matrix with a deterministic benchmark harness: every direction, every rail, one at a time and everything at once, 8 seconds per data point, reproducible run to run.

The headline picture, as a ladder from ordinary Ethernet to everything at once:

And the same data as tables, grouped by the question each layer answers.

Layer 1: raw fabric (RDMA, RAM to RAM). Does the network deliver?

Two rows are the actual proof. The latency under load: a 2-byte write getting one tenth of a microsecond slower while the same switch carries full traffic is PFC keeping the lossless queue from ever hitting a full buffer. And the TCP line: even eight tuned streams only reach wire speed by burning whole cores, while RDMA does it for free.

Layer 2: GPU traffic (NCCL, VRAM to VRAM). Does it survive the full software stack?

Point-to-point at 94.8 means the entire stack of container, torchrun and NCCL costs nothing on that pattern: wire speed, GPU memory to GPU memory. Getting every row of that table green, and understanding the two that resisted, took the detective stories below.

If you want the 9.68 Gbit/s anecdote that goes with it: one node path sat pinned at exactly that value for days. Root cause was dead MAC learning in the ASIC, the switch flooded unknown unicast instead of switching it, and the Marvell’s flood pipeline caps right there. The one command that cracked it: show mac -v 8 during a slow transfer, destination MAC not in the table, case closed. The fix escalated in three steps: writes to config_db were ignored, writes to APP_DB were ignored, and the only door that reaches the ASIC in this build is swssconfig with an FDB_TABLE JSON injected into the swss container. A static entry brought the path to 93.89 instantly; a switch reboot later healed the dynamic learning itself. Lesson: a suspiciously constant bad number means a mechanism, not noise.

Two 100G ports per node: what dual-rail actually buys you

Each node carries a dual-port 100G NIC, and the obvious expectation is “two ports = 200 G”. The measured truth is more interesting, and it is a PCIe lesson, not a network one. Here is the whole fabric with the card-level picture, PCIe geometry included:

The green node matters later: the Threadripper is the only machine where NIC and GPU sit close enough for full GPUDirect in both directions. Keep that in mind for the last chapter.

What the dual ports deliver, measured:

• One direction, both ports: 114 G, not 200. The card sits on PCIe Gen3 x8, which is ~128 G raw minus protocol overhead. Both ports share that slot budget, so the second port adds only ~20 G unidirectionally. The wires are not the limit; the slot is.
• Both directions, both ports: 214 to 220 G. PCIe is full duplex, TX and RX each get their own ~114 G budget, and suddenly the second port pays off in full. Bidirectional traffic (which is what allreduce-heavy training generates) is exactly where dual-rail shines.
• Why not LACP? A bond pins each RDMA flow to one link (one flow hashes to one port, always). Two independent rails with separate subnets let NCCL stripe across both ports itself, which it does natively, and failure domains stay clean. One caveat from testing: with a single GPU per node, dual-rail NCCL was actually slower (32 G); rails start paying once several GPUs per node feed them.
• The honest fix for the x8 ceiling is a Gen5 x16 card (ConnectX-7 class). Roadmap, not regret: the bidirectional 220 G per node pair is what the workloads actually see.

One trick worth sharing for anyone testing with a single RDMA node. Put the two NIC ports in separate network namespaces. Otherwise the Linux kernel shortcuts the traffic through loopback and it never physically hits the switch. With the ports in different netns, the traffic is forced out one port, through the switch, and back in the other. A real fabric path from a single box.

Making it survive a reboot

The catch: everything above lives in the running redis config_db, in RAM. It is not written to the on-disk config_db.json. It holds rock-solid at runtime. I watched Ubiquiti’s config enforcer daemons (ubnt-lnxcfgd, hostcfgd) run for hours without reverting the QoS tables, which is better than older firmware where people reported the config getting wiped after minutes. But a reboot wipes it.

You could config save it to disk, but that risks colliding with Ubiquiti’s own config layer (ubnt_cfg.json), and it is exactly the kind of thing a firmware update will stomp. The approach that actually holds up is a small idempotent boot hook. A systemd oneshot that re-applies the RoCE config_db entries after every boot, waiting for swss to be ready first and verifying the ASIC attribute at the end. It sidesteps the Ubiquiti config layer entirely, because it just re-asserts state after the box is up rather than fighting to persist through the boot.

That keeps RoCE reboot-proof without touching Ubiquiti’s baseline, and it degrades gracefully. If a firmware update ever changes the SONiC internals, the hook fails loudly instead of silently corrupting anything.

The fabric grows up: four nodes, and four detective stories

The switch was the beginning. Bringing four nodes onto the fabric produced its own set of lessons, each one invisible to every datasheet involved.

The node that answered pings and refused connections. After a reboot round, one node stopped accepting RDMA connections as a server; as a client it worked. Ping fine, FDB fine, GIDs fine, loopback fine. The culprit sat on a different machine: a stale ARP entry mapped the server’s rail-1 IP to its rail-2 MAC. With two NICs in the same subnet, Linux answers ARP for either port (ARP flux), and one cached wrong answer sent every connection request to the port where nobody listened. The fix, on every node:

ip neigh flush dev <rail-if>
sysctl net.ipv4.conf.all.arp_ignore=1
sysctl net.ipv4.conf.all.arp_announce=2

The reboot that undid everything. The first unplanned reboot round wiped RoCE IPs, reset MTU to 1500 (which quietly caps RDMA at 83 Gbit/s) and renamed one node’s RDMA devices entirely. Everything had been configured live, nothing persisted. The cure mirrors the switch’s boot hook: a roce-node-apply.service per node reads /etc/roce-node.conf and re-asserts rail IPs, MTU, DCB and sysctls on every boot. Since then reboots are boring, which is the highest compliment infrastructure can earn.

Firmware, levelled live. The ConnectX-5s arrived on three firmware levels, and the newer LTS notes list a RoCE retransmission fix, exactly the class of bug you do not want to chase as a phantom later. mstflint flashes with the node running, mstfwreset -l 3 activates in ten seconds without a reboot, Kubernetes nodes stay Ready. Only the virtualized node needed the host-side dance (VM stop, vfio unbind, mstfwreset --skip_driver, VM start), because a function level reset inside a guest cannot activate firmware.

Two rails, no LACP, on purpose. Each dual-port NIC feeds two independent rails: separate subnets, no bonding. LACP would pin each RDMA flow to one link; NCCL stripes across rails natively and gets the aggregate without pretending two cables are one.

From wire speed to GPU speed: NCCL and GPUDirect

The fabric being fast is necessary, not sufficient, the GPUs have to use it. That turned into its own hunt with three chapters.

The allgather detective. Allreduce did ~90 Gbit/s while 3-node allgather crawled at 18.5 on the same wires. I chased NCCL versions (built 2.30 from source: nothing) and confused myself with busbw formulas long enough to trust a measurement that was an artefact, embarrassing, and it stays in the story. What settled it was a hand-rolled isend/irecv ring doing exactly the allgather pattern: 137 Gbit/s. The fabric could do the pattern; NCCL’s algorithm choice could not. NCCL_PROTO=LL128 took 3-node allgather from 18.5 to 65.6, a factor of 3.5. Two honest footnotes from the A/B validation: pin it per job, not globally, a global LL128 pin costs 20 to 30 % on almost everything else; and the 2-node variant of this bug is fixed upstream in NCCL ≥ 2.28, so always re-test your workarounds against current versions. NCCL’s defaults are tuned for DGX-class boxes with NVLink and GPUDirect; on a consumer board they can pick pathologically wrong for specific patterns. Nothing is broken, it is tuned for a machine you don’t have.

Update: NCCL 2.30.7 (released after this was written) fixes the algorithm choice: 3-node allgather went from 18.5 to 91 Gbit/s in our re-test, no LL128 pin needed. The workaround above stays documented for older versions — and as a case study in reading busbw tables too confidently.

The alltoall that wasn’t broken at all. After everything was tuned, 3-node alltoall still collapsed to 8 to 17 Gbit/s, in our benchmark harness. The official nccl-tests did 79 to 87 on the same nodes in the same minute. Three days of suspecting the fabric and the silicon ended two lines deep in our own Python: we sized the tensor as n -= n % 3 to make it divisible by world size, which produced per-peer chunks of 178,956,968 bytes, not divisible by 16. Every peer chunk landed on a misaligned address, NCCL fell off its vectorized 16-byte path onto the element-wise path, factor 5. The official tests silently align (count & ~(16/eltSize - 1), alltoall.cu line 19), which is why they never showed it. Two elements less in the tensor: 46 G. A naturally aligned 768 MB tensor: 87.9 G. The most expensive lesson of the whole project, and it cost nothing to fix: before you blame the hardware, run the reference tool, and read how it conditions its inputs. That rule has since been promoted to law here (we call it Rule Zero).

Reading the source instead of guessing. GPUDirect refused to enable with the log line GPU Direct RDMA Disabled (distance 8 > 5). Two days of symptom-poking (forcing GDR_LEVEL=SYS: vendor error 81) achieved nothing; fifteen minutes in NCCL’s paths.cc explained everything: distance 8 is PATH_PHB, path through the CPU root complex, and the default threshold is PXB = 5, a shared PCIe bridge. The official knob is NCCL_NET_GDR_LEVEL=PHB. That ratio, minutes of code reading versus days of guessing, held for every hard problem in this story.

The phantom in barrier(). In some containers dist.barrier() crashed with CUDA unknown error, sometimes, the signature of flakiness, and flaky benchmarks are worthless. Eleven eliminations later: torch.zeros(1) after an InfiniBand collective crashes, the same call placed before it as warmup and everything runs. NGC containers default to CUDA_MODULE_LOADING=LAZY, our hand-built containers mounted an incomplete driver-library set, and lazy loading deferred the failure to the first kernel launch after NCCL init, barrier() is internally just zeros(1) plus an allreduce, so it took the blame. “Flaky” means there is a variable you are not seeing.

The read that never arrives: a silent PCIe failure

With GDR finally enabled, the two Intel Arrow Lake nodes showed a clean split that the Threadripper PRO node does not have: the NIC writing into GPU memory works (59 Gbit/s into VRAM, on every node); the NIC reading from GPU memory fails. Every time. On the Threadripper the identical software stack reads VRAM at 84 Gbit/s single-rail, 180 Gbit/s across both rails.

An earlier round had ended at “probably the silicon”, and that bothered me, elimination is not mechanism. So it got rolled up again with one rule: no verdict without positive proof. The reproducer shrank to a single machine, NIC port 1 reads GPU memory served by NIC port 0, out through the switch and back:

Deterministic, seven of seven runs. A 64-byte read fails exactly like a 128-kilobyte one; RC, DC and XRC transports fail identically; host RAM on the same path does 97.6 Gbit/s. And the software side is provably clean: bpftrace on the kernel’s dmabuf path shows the NIC being handed the exact GPU BAR1 bus address (sg0: dma=0xa000600000), every mapping call returning success. Whatever kills the read happens after software hands over to hardware.

The part that still makes me sit back: there is no error anywhere. We cleared every error register on the GPU, the NIC and both root ports, ran the failing read, and read them all back, not one bit set. No Unsupported Request, no Completion Timeout, no AER entry:

Writes are posted, fire and forget, they route fine. Reads are non-posted, they need a real answer, and between these two root ports the answer is wrong in a way no error counter considers worth mentioning. The exclusion list is long and each item was tested live, one variable at a time: IOMMU off, ACS off, NoSnoop/RelaxedOrdering toggled, driver registry switches with clean reboots, BAR1 resized from 32G to 256M (rules out the known Z690/Z790 resizable-BAR workaround class), cold and warm GPU, two perftest builds, NIC firmware config diffed byte for byte against the working node. And a telling detail from the driver source: NVIDIA’s open kernel modules keep a chipset quirk table, Z690 has an entry, Z790 has an entry, Z890 (Arrow Lake) is not in the table at all. A second Arrow Lake machine, different CPU SKU, different NIC layout, even virtualized with passthrough, shows the identical failure signature.

The evidence says the Arrow Lake root complex mishandles NIC-initiated non-posted peer-to-peer transactions between root ports, silently. We wrote it up as an upstream report, NVIDIA/open-gpu-kernel-modules#1253, with the reproducer, the bpftrace proof and the full exclusion matrix, because the next person deserves better than forum threads ending in “I gave up and bought a bigger server”. Practically the cluster barely suffers: NCCL detects the broken read path and falls back to host staging, and the numbers in the table above hold. But “works around it” and “works” are different words.

Update, one day later: NVIDIA answered — and not just anyone, but one of the GPUDirect architects. The confirmation is as short as three days of debugging deserve: “Intel client CPUs are known to only support PCIe P2P writes.” It is a fundamental hardware limitation of the client root complex, there are no driver knobs, and the only real workaround is putting GPU and NIC behind a shared PCIe switch. So: not a bug in our setup, not a driver regression — silicon. Every workstation and server platform (Xeon W, Threadripper, EPYC) routes P2P reads fine; Intel’s client dies simply don’t. The one thing that still stings: none of this is written in any public Intel datasheet. Now it is at least written here.

When to do this, and when to just buy the switch

I am not going to pretend this is a supported production setup. It isn’t. So, honestly:

Do this if you have one of these switches already, you are running a homelab or a small cluster, and you would rather spend a weekend understanding your fabric than a few thousand euro on another box. You learn an enormous amount about how RoCE actually works, and you get a genuinely lossless fabric out of hardware you own.

Buy the real switch if this is a business-critical fabric, you need vendor support, you are past a handful of nodes, or you cannot tolerate the reboot-persistence hack and the total absence of a safety net. A proper Mellanox/Spectrum switch gives you native, supported, plug-and-play RoCE with SHARP and the rest. At scale, that is worth the money.

For 12 GPUs across 4 nodes, doing inference and the occasional multi-node training run, the switch I already owned is the right answer. Your line will be somewhere else. Draw it honestly.

An open ask to Ubiquiti

Ubiquiti, the hardware already does this. It’s SONiC, it’s a Marvell ASIC on the PFC-capable list, the lossless pool is already carved out in the buffer config. The only thing standing between your Enterprise Campus Aggregation line and native RoCE is a pg_profile_lookup.ini, the lossless profiles, and a handful of QoS toggles in the UI. You are withholding a feature the silicon ships with.

That is not a cost saving. It’s a missed selling point. “Lossless RoCE for GPU and storage fabrics, on a switch that costs a fraction of a Mellanox Spectrum” would sell itself to every homelab, every small AI shop, every university group standing up a GPU cluster on a budget. That is a fast-growing market and it is shopping for exactly this. Expose PFC/DCB natively, even as an advanced or unsupported toggle, and you own a segment that currently has to either overspend or do what I just did. Please build it in.

Takeaways

• The UniFi Enterprise Campus Aggregation switch runs SONiC on a Marvell ASIC. It can do lossless RoCE, the capability just isn’t exposed.
• Never config qos reload it globally. That crashes the control plane. Recover with a reboot; it reloads the clean on-disk config.
• Enable RoCE port-by-port via config_db, build the missing lossless buffer profile yourself, use [TABLE|key] references, and verify at the ASIC (PRIORITY_FLOW_CONTROL = 24).
• Config lives in RAM. Make it reboot-proof with an idempotent systemd boot hook, not config save.
• Works with ConnectX and Intel E810/E835 on the NIC side, for the lossless fabric. For real GPUDirect (NIC writing straight into VRAM), ConnectX it is. Carries training as well as inference.
• Dual-port 100G: the slot decides, not the cable. On Gen3 x8 the second port pays off bidirectionally (220 G), barely unidirectionally. Independent rails beat LACP for RDMA.
• A suspiciously constant bad number means a mechanism, not noise. And before you blame hardware: run the reference tool and read how it conditions its inputs. That rule (we call it Rule Zero) was learned the expensive way, via two misaligned tensor elements.
• Minutes of source reading beat days of symptom-guessing: paths.cc for the GDR distance, alltoall.cu for the alignment, ProcessGroupNCCL.cpp for barrier().
• Everything configured live dies on reboot. Idempotent apply-services everywhere: a boot hook on the switch, roce-node-apply on every node.
• When a platform silently cannot do something (Arrow Lake: NIC reads from GPU memory), prove it, write it up, report it upstream: #1253.
• Know where your line is between “unlock what I own” and “buy the supported box.”

Update, July 22 — we measured the theory to death anyway. NVIDIA’s statement above stood, but so did one suspicion: our slow node reported MaxPayload 128 bytes at the GPU root port instead of 256 — in the x8 bifurcation split, while the x16 boards showed 256. A BIOS changelog entry (“Silicon Initialization Code” = an Intel FSP bump) smelled like a fix. So: flashed the BIOS fully remotely, through the BMC, no hands on the machine — the BMC’s official pull mechanism was broken by the way (it simply never fetches the image), the only thing that worked was multipart push with an undocumented OemParameters={"ImageType":"BIOS"} and the host powered off first. Afterwards, with incidents (the flash wipes the Secure Boot MOK list → the kernel rejects the NVIDIA DKMS module; and the old NVRAM settings profile forced 128 again — only a CMOS clear via IPMI made the 256 stick): root port reports DevCap 256, GPU negotiates 256, the whole chain clean. The PCIe init bug was real and is fixed.

And the bandwidth? 5,203 MiB/s. The exact same number as before, to the decimal. Message size sweep from 2 bytes to 8 MiB: hard ceiling from 1 KiB up. Four QPs instead of one: zero scaling. The complete official tuning checklist (NVIDIA GPUDirect docs + Mellanox/HPCAC): MAX_ACC_OUT_READ was already at the recommended 44, set PCI_WR_ORDERING=force_relax in firmware and activated it live via mstfwreset — no change. ACS off, IOMMU off (zero groups, verified), rBAR 32G, MTU, all of it. The ceiling does not move by a single Gbit.

The lesson in one sentence: MaxPayload 128 was a real bug, but not our bottleneck — the P2P write path through the Arrow Lake root complex is hard-capped too, 43.6 Gbit/s in our case, no matter what the config registers say. The correlation (slow node = the only one showing 128) looked like proof and was coincidence. The A/B test that exposed it cost one BIOS flash. Epistemology doesn’t come cheaper.

A by-catch that might save someone days: the GeForce RTX 5090 in the fourth node reports CU_DEVICE_ATTRIBUTE_DMA_BUF_SUPPORTED = 0 — same driver, same kernel as the RTX PRO cards next to it, which report 1. Driver update, IOMMU toggle, kernel update: stays 0. GeForce is simply excluded from GPUDirect RDMA, product policy since forever (“available on Tesla and Quadro GPUs”), and the old nvidia-peermem fallback doesn’t even load with the open kernel modules anymore. If you want GDR: RTX PRO and up, and no Intel client silicon. Our fabric numbers above are untouched by this — host staging still carries 94G wire speed on unidirectional patterns.