A scheduler that miscounted

- 9 mins read

One of our GPUs runs two things at once. A text-to-speech service and a worker, sharing the same physical card through MPS. That is on purpose: together they fill one card cleanly, and split apart each would waste one.

Sounds solved. It wasn’t. The scheduler counted that one card twice, put the node at used: 2, capacity: 1, and with that shut it down for every further pod, including ones that wanted no GPU at all. A node with one free card, claiming it has none.

This is the story of how I found that, proved it in the cluster, fixed it in prod, and got the fix back into the upstream project. And the part where an AI review bot found a mistake my own tests had waved through.

Why two pods share a GPU in the first place

Quick on the starting point, because it explains the rest. Our cards are scarce and expensive: Blackwell for training, a handful of Pro cards for the rest. Not every workload needs a whole GPU. The TTS service and its worker together fit onto one card comfortably, if you let them share.

The tool for that is Kubernetes’ Dynamic Resource Allocation plus the NVIDIA DRA driver. Instead of “give me a GPU” you write a ResourceClaim, and several pods can reference the same claim. Through MPS or time-slicing they then share one physical card. The claim status carries reservedFor with both consumers in it.

The cluster runs on a GPU-aware scheduler: the KAI scheduler, originally from NVIDIA, built for exactly this kind of AI workload. It knows queues, fair-share, GPU topology. A solid piece of software. It just had a blind spot on the shared claim.

The bug

The scheduler keeps a per-node vector of how many GPUs are in use. When a pod is scheduled, it adds that pod’s GPU demand to this used vector. But with a shared claim, every consuming pod carries the same physical card in its allocation, and the scheduler added it once per pod.

Two pods on one card makes used: 2 in that math. The card exists only once, so capacity: 1. The free remainder is computed as capacity - used, so 1 - 2 = -1. Negative.

And a node with a negative idle value gets rejected by the resource check for everything. Not just GPU pods. Even a pod requesting zero GPUs, because 0 > -1 already trips the check. From the outside it looks like this:

<node>: Node didn't have enough resources: GPUs,
        requested: 0, used: 2, capacity: 1

A pod that wants no GPU gets turned away because supposedly there is no GPU, on a node that has one. That is the kind of error message you read three times and then start not trusting the log anymore.

Prove it first, then touch it

A rule here: nobody experiments on the production scheduler. So the proof had to go into a corner where it can’t knock anything over.

I stood up a second scheduler next to it, own name, own deployment, not managed by our GitOps. It only binds pods that explicitly ask for it, and cannot touch the real workloads at all. Then I rebuilt the failure case exactly: a shared claim on a free card, two consumers on it, both bound, reservedFor with both names. Precisely the state of the voice pipeline.

And then the actual A/B test: the same scheduler, once with the stock image, once with my fix, nothing else changed.

  • Stock: a GPU-less test pod stays Pending, in the log verbatim the used: 2, capacity: 1 line from above.
  • Fix: the same pod runs. The claim still has two consumers, the card is now counted once instead of twice.

The road there had a few stumbling blocks nobody documents because they are too mundane: the custom scheduler doesn’t get its PodGroups automatically, the test queue has zero quota, the shared claim needs a specific label. Every single hurdle cost half an hour and was written down nowhere. It is now, on our side.

The fix

At its core a simple thing: the node now keeps a reference count per physical device, addressed by driver/pool/device from the allocation. The first pod that brings a device counts it. Every further pod that already sees the device referenced does not count it again. On removal the same in reverse: the card stays counted as long as one consumer remains, and is released only when the last one leaves.

Exclusive claims and non-DRA GPUs are left alone. The dedup only ever skips a device that another pod on the same node already contributed.

We rolled it out in prod through a dedicated image, clean over GitOps, with a one-line way back should it act up. Since then the node that was dead before runs the GPU-less probe pod through, and the voice pipeline notices none of it. That is exactly how a fix should feel: invisible.

The part that hurt

Done, for me, means upstream too. A fix that lives only in our cluster is a fork I maintain forever. So: opened an issue, filed a pull request, clean by the project’s rules. Own commit, proper description, tests that show the bug red without the fix and green with it.

And then an AI review bot the project runs on every PR found a real mistake in my fix. One I didn’t have on my radar.

There is a kind of internal pod, reservation pods, whose GPU demand the scheduler zeroes out beforehand. If such a pod references the same shared claim, my dedup would have subtracted from that zero and driven the used value negative. So the same bug as at the start, just from the other side. My tests hadn’t covered that path, because I had built against an old release and simply didn’t have the case in mind.

I added the guard, wrote a test for it, and on the first try it was green without the fix in place. Which means the test tests nothing. That is the most dangerous kind of test, because it gives a good feeling and is empty. The cause was a bug in my test setup, not in the fix: I had put the pod in a different namespace than its claim, and the lookup is namespace-local, so the code path under test never ran. Fixed, and then the test went red for the right reason and green with the fix.

What I take from it

Two things, concrete.

Build against the branch you file the PR against, not against the release you happened to check out. The whole reservation case and one more convention mistake came from starting on an old tag while the rules had moved on since.

And: a green red-test is not a passing test, it’s a broken setup. A test that can’t go red is worse than no test. It’s self-deception with a green checkmark.

On the question whether such a review bot is worth it, since it’s right there: it caught a real edge case I had missed. A second pair of eyes that never gets tired catches exactly the cases you are blind to by your own assumptions. But it wouldn’t have seen the convention mistake, that came from reading the project docs, and one of its notes was plain noise. As one stage in the process it’s worth something, as a replacement for the craft before it, nothing.

The node counts right now, the PR sits with the project, and the one test that could have lied before can’t anymore. That’s enough for today.

For those who want the details

The story ends here. From here on it’s code. If you want to know exactly where the counter hangs and why it looks the way it does, read on.

The entry point is addTaskResources in node_info.go. The function takes a task’s resource demand and adds it onto the node’s vectors: CPU, memory, GPU. For GPUs that demand came from the pod’s DRA allocation, and that’s exactly where the bug lived: every pod consuming a shared claim carries the same physical device in its allocation. Two pods, two allocations, the same device twice, added twice.

My change sits one line after the demand vector is built:

func (ni *NodeInfo) addTaskResources(task *pod_info.PodInfo) {
    // ... build demand vector ...
    ni.dedupSharedDRAGpus(task, resourcesToTrackVector)
}

The node now keeps a new map, DRASharedDeviceRefCount, a reference counter per physical device. The key is whatever uniquely identifies a device in the DRA model:

func draDeviceKey(result resourceapi.DeviceRequestAllocationResult) string {
    return result.Driver + "/" + result.Pool + "/" + result.Device
}

Driver/Pool/Device: the driver, the pool the device comes from, and the device name within it. Two pods sharing the same card produce the same key. That’s exactly what I want: a stable fingerprint of the physical card, regardless of how many pods hold it right now.

The dedup itself is then almost boring. The first pod to bring a device counts it, every further pod sees the reference and leaves it alone:

alreadyCounted := 0.0
for _, key := range ni.allocatedGPUDeviceKeys(task) {
    if ni.DRASharedDeviceRefCount[key] > 0 {
        alreadyCounted++
    }
    ni.DRASharedDeviceRefCount[key]++
}

allocatedGPUDeviceKeys filters to GPUs beforehand: only devices whose driver is the NVIDIA DRA driver. Exclusive claims, CPU devices, everything else the dedup never touches. That mattered to me: the change should be surgical, not rewrite half the accounting.

It gets interesting at the edges, and one of them is exactly what the review bot found. Before the dedup there’s a line that looks like a triviality:

current := resourcesToTrack.Get(resource_info.GPUIndex)
if current <= 0 {
    return
}

It’s not cosmetic. Reservation pods carry zero GPUs, the scheduler zeroes their GPU index beforehand. Without this return the dedup would subtract something from that zero and drive the used value negative. The same bug as at the start, just from behind. And even when the value is positive, a second line caps the deduction:

if alreadyCounted > current {
    alreadyCounted = current
}

Never deduct more than the task contributed itself. Two lines that in the normal case never fire, and that’s exactly why I didn’t have them on the first pass. The release path, releaseSharedDRAGpus, mirrors the whole thing in reverse: reference down, and only when a device’s counter hits zero does the card get released in the vector again. Symmetry, otherwise the state drifts apart over time.

That’s the whole fix. One counter, a key from three fields, two guards at the edges. The hard part was never the code. It was seeing the case at all.