You end this with a GPU visible inside a container scheduled through Dynamic Resource Allocation (DRA), a one-line check that tells this failure apart from every "your driver is broken" answer on the internet, and a systemd unit that stops the node re-arming the bug at the next reboot.

Short version: if nvidia-smi works on the host but the pod prints NVIDIA-SMI has failed, grep the claim's Container Device Interface (CDI) spec under /var/run/cdi for /dev/nvidia. A count of zero means the kubelet plugin generated and cached a spec with the NVIDIA libraries and no device nodes. Run nvidia-modprobe -c 0 -u on the host, then delete the pod and claim and restart the plugin DaemonSet so the spec is regenerated.

Why nvidia-smi works on the host and dies in the pod

The failure looks like this inside the container, quoted from the report in kubernetes-sigs/dra-driver-nvidia-gpu issue #1380:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Everything around it reports success. The ResourceClaim says allocated,reserved, the pod reaches Running, the kernel module is loaded, and nvidia-smi on the node lists the cards. Nothing crashed, and the GPU is not in the container.

That string has a well-known set of causes on a normal host: a failed DKMS rebuild, Secure Boot blocking the module, a userspace/kernel version mismatch, a card that fell off the PCIe bus. None of them apply here, because the host driver is fine. What broke is the CDI spec the kubelet plugin wrote for this specific claim: it carries the NVIDIA userspace libraries and no /dev/nvidia* device nodes, so the NVIDIA Management Library (NVML) inside the container finds the tooling on $PATH and no hardware to talk to. Same error text, different layer, and the split is the same shape as Xid 154 routing four different remediations through one log line: the symptom is a pointer, not a diagnosis. If the node also does GPU passthrough, settle the VFIO and DRA overlap before you start, because a card bound to vfio-pci fails earlier and differently.

Node boots, nvidia kernel module loadsPod scheduled, kubelet plugin runs NodePrepareResourcesPlugin generates a CDI spec for the claimDo /dev/nvidia* exist yet?Spec lists deviceNodes, container sees the GPUSpec has library mounts only, zero deviceNodesSpec cached under /var/run/cdi plus checkpoint.jsonPod reaches Running, nvidia-smi exits 9yesno

What you need before you start

  helm install dra-driver-nvidia-gpu \
    oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
    --version 0.5.0 \
    --create-namespace \
    --namespace dra-driver-nvidia-gpu \
    --set gpuResourcesEnabledOverride=true
  • A root shell on the GPU node. Several steps read /var/run/cdi and /var/lib/kubelet, which the API server cannot show you.
  • An NVIDIA kernel driver on the host. The issue used 580.105.08 on RTX PRO 2000 and RTX PRO 4000 Blackwell cards; nothing about the mechanism is card-specific.
  • kubectl able to read resourceclaims and delete pods in the workload namespace.

One version check first. As of 15 September 2026, issue #1380 is open against v0.5.0 and carries the v0.5.1 milestone, which has not shipped. Look at the releases page before you do any of this: if v0.5.1 is out by the time you read it, upgrade first and use the recovery below only for nodes already stuck.

Read the claim's CDI spec, not the pod logs

1. Confirm the claim really was allocated. Rule out the boring case, a pod that never got a claim.

kubectl get resourceclaims -n "$NS"
kubectl get resourceclaim -n "$NS" "$CLAIM" \
  -o jsonpath='{.status.allocation.devices.results}' | jq .

Empty status or a Pending pod is a scheduling problem and stops here. allocated,reserved plus a Running pod is the signature you are chasing.

2. Ask the host whether the device files exist.

ls -l /dev/nvidia*

On a fresh headless node that has never run an NVIDIA userspace client, this returns No such file or directory, which is the core observation in issue #1380. The module is loaded and the character devices were never created. nvidia-modprobe exists for exactly this reason: creating the device files is normally a distro or udev job and is not guaranteed to have happened, which is why the tool is installed setuid root to do it on demand (nvidia-modprobe(1)).

3. Count the device nodes in the spec. This is the command that splits the two failure modes.

CLAIM_UID=$(kubectl get resourceclaim -n "$NS" "$CLAIM" -o jsonpath='{.metadata.uid}')
grep -c '/dev/nvidia' "/var/run/cdi/k8s.gpu.nvidia.com-claim_${CLAIM_UID}.yaml"

Name the variable something other than UID: bash marks UID readonly, and the assignment fails in an interactive root shell before you ever reach the grep. A healthy spec lists /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm and /dev/nvidia-uvm-tools under containerEdits.deviceNodes, the structure NVIDIA documents for CDI-generated specs (NVIDIA CDI support). A count of 0 while the library mounts are still present is your answer: the container received the userspace stack and no hardware, so nvidia-smi exits 9, the NVML code for driver-not-loaded.

Three ways a claim ends up with no device nodes

  • The node has never had them. Headless machine, no X server, no container that ran nvidia-smi, no udev rule doing the work at boot. The plugin prepares a claim against a /dev that has no NVIDIA entries in it.
  • The node rebooted. The device files are not persistent across boot on a headless host, so a node that worked yesterday can come back with the same empty /dev and re-arm the bug on the next pod.
  • Something created them afterwards. Once the plugin has prepared a claim, the spec is cached, so device files that appear later do not reach that claim. This is why the bug is a race and not a static misconfiguration, and why the timing looks random across a node pool.

Recover the stuck pod, in this order

4. Create the device nodes on the host.

nvidia-modprobe -c 0 -u
ls -l /dev/nvidia*

-c 0 creates the device file for minor number 0 and -u loads nvidia-uvm and creates its node. Running nvidia-smi -L has the same side effect, which is how "it started working after I SSH'd in to check" becomes such a misleading bug report on this failure.

5. Invalidate the cached spec. Step 4 on its own does not fix the running pod, because a prepared claim is tracked in two places the host has no idea about: the per-claim spec under /var/run/cdi, and a PreparedClaims entry in checkpoint.json under /var/lib/kubelet/plugins/gpu.nvidia.com/.

kubectl delete pod -n "$NS" "$POD"
kubectl delete resourceclaim -n "$NS" "$CLAIM"   # skip if it came from a ResourceClaimTemplate
kubectl rollout restart daemonset/dra-driver-nvidia-gpu-kubelet-plugin \
  -n dra-driver-nvidia-gpu
kubectl rollout status daemonset/dra-driver-nvidia-gpu-kubelet-plugin \
  -n dra-driver-nvidia-gpu

Delete the workload first so NodeUnprepareResources runs and removes the stale spec instead of orphaning it.

6. Recreate the workload and repeat step 3 against the new claim UID. The grep count should now be non-zero.

Make the node create its device nodes before kubelet starts

7. The durable fix is ordering, since the bug is a race. A oneshot unit that runs ahead of kubelet:

# /etc/systemd/system/nvidia-device-nodes.service
[Unit]
Description=Create NVIDIA character device nodes before kubelet starts
After=systemd-modules-load.service
Before=kubelet.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-modprobe -c 0 -u
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now nvidia-device-nodes.service

On multi-GPU nodes, repeat -c per minor number. Put this in the node image or the bootstrap script rather than a runbook, because a fix that depends on someone reading a runbook after a reboot is not a fix. I would not assume nvidia-persistenced covers this for you either: its own startup expects the device files to exist, so verify against the driver version you actually run before you drop the unit.

Verify the container sees the GPU

kubectl exec -n "$NS" "$POD" -- nvidia-smi -L

Expect one GPU 0: NVIDIA ... (UUID: GPU-...) line per allocated device and exit status 0. Then check the container's view against the claim:

kubectl get resourceclaim -n "$NS" "$CLAIM" \
  -o jsonpath='{.status.allocation.devices.results[*].device}'

The device count from the claim and the line count from nvidia-smi -L have to agree. On the host, nvidia-ctk cdi list should enumerate the nvidia.com/gpu=* devices, confirming the toolkit itself sees hardware.

Where this goes wrong

  • Running nvidia-smi on the host and declaring victory. It creates the device files and changes nothing for an already-prepared claim, because the cached spec is not regenerated. The pod keeps failing and the operator concludes the driver is flaky.
  • Working the generic fix list. DKMS rebuilds, nvidia-uvm reloads, Secure Boot checks and driver reinstalls all target a broken host driver. If nvidia-smi -L works on the node, every one of them is wasted time, and grep -c '/dev/nvidia' on the claim spec is what tells you so in one second.
  • Misfiling a UVM-only gap. If /dev/nvidiactl and /dev/nvidia0 are present but /dev/nvidia-uvm is missing, nvidia-smi succeeds and CUDA initialization fails instead. Different symptom, different fix, and the -u flag in step 4 is what covers it.
  • Blaming the kubelet. The plugin ships as a DaemonSet, so node-level state surfaces as a workload failure two layers away from its cause, the same indirection as the kubelet's cgroup v1 check misreading a v2 host.

FAQ

Why does nvidia-smi work on the host but not in the pod? The host has a working driver and the container has a CDI spec with no device nodes in it. The binary and NVML are mounted in, the hardware is not.

Does restarting the pod fix it? Only if the claim is deleted too. A restart against the same prepared claim reuses the cached spec under /var/run/cdi and fails identically.

Where is the CDI spec for a DRA claim? /var/run/cdi/k8s.gpu.nvidia.com-claim_<claim-uid>.yaml, with a matching PreparedClaims entry in checkpoint.json under /var/lib/kubelet/plugins/gpu.nvidia.com/.

Is this fixed in a released version? Not as of 15 September 2026. Issue #1380 is open against v0.5.0 with the v0.5.1 milestone attached, and v0.5.1 has not shipped. Check the releases page before you assume you still need the workaround.

One grep tells you which failure you have

The whole diagnosis is grep -c '/dev/nvidia' against the claim's CDI spec. Zero means the spec is device-less and the host driver is a red herring. Non-zero and a still-broken pod means you have an actual driver problem and the generic advice applies after all. Everything above is what you do on either side of that number.