NVIDIA GPU Operator 26.7.0 shipped on 21 August with the donated DRA driver as a managed component, and buried in the release notes is a sentence that should stop a platform team mid-rollout: with passthrough enabled, one physical GPU is published to the scheduler twice, and kube-scheduler can grant both claims.

Short version: turn on DRA VFIO passthrough and every eligible GPU is advertised as a container device under gpu.nvidia.com and as a passthrough device under vfio.gpu.nvidia.com. The driver only notices the overlap during Prepare, which runs after the scheduler has committed, so a container claim and a VM claim for the same card can both succeed. NVIDIA suggests submitting workloads sequentially. Dedicate the nodes instead.

What the 26.7.0 release notes actually admit

The wording in the GPU Operator 26.7.0 release notes is unusually direct: "With DRA passthrough enabled, the driver initially advertises each eligible physical GPU as both a container GPU and a VFIO device. If a container GPU claim and a VFIO passthrough claim for the same physical GPU are allocated before either device is prepared, the scheduler can allocate both claims."

That paragraph is the complete public documentation of the problem. It describes the symptom and stops. The mechanism underneath it lives in a scheduling KEP that changed shape between Kubernetes 1.35 and 1.36, which is why an assistant asked about this today will answer from the device-plugin era, when a GPU had exactly one owner because there was exactly one nvidia.com/gpu integer to decrement.

Context for how fast this spreads: NVIDIA donated the GPU DRA driver to the Kubernetes community at KubeCon Europe 2026 in Amsterdam, and it now lives at kubernetes-sigs/dra-driver-nvidia-gpu. Release 26.7.0 promotes it to a first-class operand next to driver 595.91.07 and container toolkit v1.20.0. Teams that were waiting for DRA to feel supported are about to enable it. This is the same pattern as the mirror pod secret rejection in 1.37: a node-level component is the only thing that knows the truth, and it learns it too late to say no.

Why kube-scheduler cannot catch this

DRA splits a device lifecycle into three stages. The kubelet plugin publishes ResourceSlices describing what exists on the node. kube-scheduler allocates claims against those slices, cluster-wide, with no node-local knowledge beyond what was published. Then NodePrepareResources runs on the node and produces the CDI spec that actually wires the device into the workload.

Physical exclusivity is only knowable at stage three. The binding decision happened at stage two.

Physical GPU 0node gpu-a1ResourceSlicedriver gpu.nvidia.comResourceSlicedriver vfio.gpu.nvidia.comkube-schedulerallocates from slicesPrepareCDI spec for /dev/nvidia0Preparebind vfio-pci, /dev/vfio/devices/vfio0One GPU, two owners"container claim""VFIO claim"

Two claims, two Prepare calls, one card. The container path opens /dev/nvidia0. The VFIO path unbinds the nvidia kernel driver from the PCI function and binds vfio-pci. If a training job already holds the character device open, that unbind either returns EBUSY or tears the device out from under a running process.

The asymmetry is the part I would plan around. The container job dies loudly and someone files a ticket. The VM boots against a card whose teardown path never completed. Whether framebuffer contents survive that exact sequence is a test I would run on a spare node before trusting a tenancy boundary to it, and I have found no published result either way. If you sell GPU capacity to more than one tenant, that is now a boundary you own, in the same family of questions as running workloads under user namespaces in 1.36 and default-deny egress between tenants.

Seeing the VFIO DeviceClass is not evidence you are exposed

Here is where I expect the panic to be wrong. The PassthroughSupport feature gate is Alpha and off by default in the DRA driver, and that single fact is what keeps this from being a fleet-wide incident this week.

NVIDIA's DRA install documentation notes that the passthrough capability "is always enabled, although VFIO passthrough is unusable without the Alpha PassthroughSupport feature gate." So vfio.gpu.nvidia.com shows up in kubectl get deviceclasses on clusters that have no passthrough at all. Someone on your team will screenshot that list and call it a finding. Check the gate on the kubelet plugin before you page anyone.

Your 2023 pod manifest may already be a DRA claim

The second-order effect is nastier than the headline, and it has nothing to do with VMs.

DRAExtendedResource (KEP-5004) is enabled by default from Kubernetes v1.36.0. It makes the scheduler translate a plain resources.limits: nvidia.com/gpu: 1 into an auto-created ResourceClaim against whichever DeviceClass carries spec.extendedResourceName: nvidia.com/gpu. A manifest written three years ago and never touched since is now flowing through the DRA allocation path. Nobody opted in, and nothing in the pod spec changed.

That translation has already produced a leak of its own. kubernetes/kubernetes#134311, filed against v1.34.0 with DRAExtendedResource: true, reported scheduler-created claims still sitting in allocated,reserved after their pods had completed. It is closed and fixed, and it marks where the sharp edges are on this path: claims whose lifecycle is owned by the scheduler rather than by you.

Kubernetes already has the primitive that would close this

KEP-4815 Partitionable Devices exists for exactly this shape of problem. It adds sharedCounters and consumesCounters to ResourceSlice so a driver can model overlapping views of one piece of hardware: allocate one device, and the counters it consumes stop being available to its siblings. That is how MIG profiles express that two partitions want the same memory slice.

Per the KEP README, DRAPartitionableDevices graduates to beta in 1.36, and a backwards-incompatible revision between 1.34 and 1.35 now requires sharedCounters to live in a separate ResourceSlice from the devices consuming them, in the same pool with matching generation. A driver migrating across that break has a defensible reason to ship the overlap unmodeled and write a known-issue paragraph. It still leaves operators holding the risk.

Find double-advertised GPUs in your own cluster

The condition is invisible in kubectl describe node. Node capacity looks normal because neither driver is reporting an integer count any more. What you are hunting is one GPU UUID appearing under two driver names:

kubectl get resourceslices -o json | jq -r '
  .items[] | .spec.driver as $d | .spec.nodeName as $n |
  .spec.devices[]? | [$n, $d, .name, (.attributes.uuid.string // "-")] | @tsv' \
  | sort -k4 | uniq -f3 -D

Any UUID on two rows with different drivers is double-advertised. Then confirm the gate rather than inferring it from the DeviceClass list:

kubectl get deviceclass gpu.nvidia.com -o jsonpath='{.spec.extendedResourceName}{"\n"}'
kubectl -n gpu-operator get pods -l app=nvidia-dra-driver-gpu-kubelet-plugin \
  -o jsonpath='{.items[0].spec.containers[0].args}' | tr ',' '\n' | grep -i feature

The first command tells you whether legacy nvidia.com/gpu manifests are being translated into claims. The second tells you whether passthrough is genuinely live.

Before you turn on PassthroughSupport

  1. Split the node pool first, at the provisioner level. Label passthrough nodes and taint them (nvidia.com/gpu.workload=vfio:NoSchedule), then make the container DeviceClass selector exclude them. Encode it in the Karpenter NodePool or node-group config so it survives autoscaling. Typed GPU pools strand capacity, and on a card this expensive that hurts. Take the strand until DRAPartitionableDevices closes the gap at the scheduler.
  2. Reject sequential submission as a control. NVIDIA's first workaround asks you to hand-serialize a system built to avoid serializing. It survives a demo. It does not survive one Argo burst or one Karpenter node coming online with two pending pods.
  3. Run the ResourceSlice UUID query on every cluster already on 26.7.0. A collision means a live allocation race, today, with no alert attached to it.
  4. Audit leaked claims if you are on Kubernetes 1.36 or later. List ResourceClaims whose ownerReferences point at a Pod in Succeeded or Failed while status is still allocated. Those are GPUs you are billed for and cannot schedule, per the failure mode in issue #134311.
  5. Pin the stack explicitly. GPU Operator 26.7.0, driver 595.91.07, container toolkit v1.20.0, on Kubernetes v1.34.2 or later, which the DRA driver requires for the resource.k8s.io DeviceClass API. Version drift here changes which of the above applies.
  6. Set a watch on DRAPartitionableDevices reaching beta in your distro. Once the NVIDIA driver models the container and VFIO views as counter-consuming siblings in one pool, the scheduler enforces exclusivity and the typed node pools become an optimization rather than a safety mechanism.

FAQ

Does GPU Operator 26.7.0 double-allocate GPUs by default? No. PassthroughSupport is Alpha and off by default, so the VFIO view of the card is unusable until you enable the gate. The DeviceClass is registered either way.

How do I tell if a GPU is advertised twice? Group ResourceSlice devices by their uuid attribute across driver names. A UUID under both gpu.nvidia.com and vfio.gpu.nvidia.com on one node is double-advertised.

Why can kube-scheduler not prevent this? It allocates at stage two of publish, allocate, prepare. The driver reconciles the two views during NodePrepareResources at stage three, after the binding decision is already made.

Does an old manifest using nvidia.com/gpu limits go through DRA now? On Kubernetes 1.36 and later it can. DRAExtendedResource is on by default from v1.36.0 and auto-creates a ResourceClaim against the DeviceClass carrying spec.extendedResourceName: nvidia.com/gpu.

What breaks first if both claims land? The container job, when the nvidia driver unbind hits an open /dev/nvidia0 and returns EBUSY or pulls the device away mid-run. The VM starts against a card whose teardown never finished cleanly.