Fix "Xid 154, GPU recovery action changed" on GPU Nodes

An inference node loses a pod, the alert fires with Xid 154 in the body, and the runbook you reach for depends entirely on a hex value most people scroll past. This walks through reading that value, acting on the tier it names, restoring GPU capacity Kubernetes will not restore on its own, and fixing the exporter default that deleted the Xid you actually needed.

Short version: Xid 154 is informational and reports a recovery tier, per NVIDIA's Xid catalog. The fault is the Xid printed on the line before it. The payload is the hex after to: 0x1 means nvidia-smi -r clears it, 0x2 means the node has to reboot.

What is Xid 154 actually telling you?

Here is the shape of the burst, from dmesg on a node that dropped a GPU:

NVRM: Xid (PCI:0000:05:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:05:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)

Two lines, two jobs. Line one is the fault (79, GPU fell off the bus). Line two is the driver stating what it will take to clear it. The same 154 string appears for a transient fault that a device reset fixes and for a node that has to go down, so the number itself routes nowhere.

What the manual actually says: the nvidia-smi documentation describes GPU Recovery Action as an "Action to take to clear fault that previously happened. It is not intended for determining which fault triggered recovery action." NVIDIA is telling you in its own field description that this is a remediation pointer with no diagnostic content. Treat it as routing metadata, the same way you treat a Kubernetes taint: it says what to do, never why.

That routing matters more as the box gets denser. Losing one GPU out of eight on a node you are paying reserved-instance money for is a real line item, which is the same arithmetic behind AI compute costs in 2026, and it is worth twenty minutes of runbook to avoid rebooting a node that needed a device reset.

Prerequisites

• A Kubernetes cluster with NVIDIA GPU nodes, the GPU Operator or a standalone k8s-device-plugin, and dcgm-exporter deployed.
• NVIDIA driver R570 or newer. The GPU recovery action field and Xid 154 arrive with that driver generation. Confirm with nvidia-smi --version.
• Node-level shell (SSH, nsenter, or a privileged debug pod). dmesg is not reachable from the workload side.
• kubectl with cordon and drain rights, plus the ability to reboot or power-cycle the machine.

The public reproduction I lean on for the hard cases is dbirks/home-k8s issue #46: an RTX PRO 6000 Blackwell (GB202, 96 GB) on Talos Linux 1.12.6, kernel 6.18.18, open kernel module 580.126.20, running vLLM v0.20.1. That is someone else's hardware and it is documented in public, which makes it checkable.

Step-by-step

1. Pull the whole Xid burst, not the line your alert quoted.

sudo dmesg -T | grep -i "NVRM: Xid" | tail -20

Xid 154 never travels alone. Take twenty lines rather than one because a single fault emits a chain: 62 (internal micro-controller halt), then 79, then 154. Grepping NVRM: Xid is NVIDIA's own documented approach in "Working with Xid Errors".

2. Ask the driver for the live tier.

nvidia-smi -q | grep -iE "GPU Recovery Action|Product Name|Minor Number"

dmesg is history. This is current state. A GPU that logged 0x2 an hour ago and has since been rebooted reports None here, and you get one line per device so you can map the tier to a minor number before you touch anything. If the field is missing entirely, the driver predates NVML_FI_DEV_GET_GPU_RECOVERY_ACTION and step 1 is your only source of truth.

3. Find out what Kubernetes already did, which is usually less than you assume.

kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl -n gpu-operator logs -l app=nvidia-device-plugin-daemonset \
  --tail=300 | grep -E "XidCriticalError|marking (device|it) as unhealthy"

The plugin's health checker watches the NVML event stream and logs XidCriticalError: Xid=%d on Device=%s; marking device as unhealthy before pulling that GPU out of allocatable. Its hardcoded ignore list is 13, 31, 43, 45, 68 and 109, visible in internal/rm/health.go. Those are application-level faults it refuses to charge to the node. A page-fault storm from one bad kernel will therefore never move allocatable, while a single Xid 79 quietly takes a node from 8 GPUs to 7 with no event on the node object. If you also run device allocation through DRA, check the DRA path where one GPU can land in both a VM and a pod, because the allocatable count is not the only accounting that can drift out of sync with the hardware.

4. Act on the tier, and only on the tier.

For 0x1:

kubectl cordon "$NODE"
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
kubectl delete pod <pod-holding-the-gpu>
sudo nvidia-smi -r -i 3

The reset refuses to run while anything holds the device, and the common surprise is that deleting the pod is not sufficient: containerd can still have /dev/nvidia3 open while the pod sits in Terminating, and nvidia-smi -r reports the GPU is in use. Query the compute apps first, wait for the pod to leave the API, then reset.

For 0x2:

kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=15m
sudo systemctl reboot

5. Give kubelet its capacity back.

kubectl -n gpu-operator delete pod -l app=nvidia-device-plugin-daemonset \
  --field-selector spec.nodeName="$NODE"

The plugin does not re-query a GPU once it has marked it unhealthy, so allocatable can sit at 7 long after the hardware is fine. This is reported in k8s-device-plugin issue #1014 and closed as not planned, which is what makes it a permanent step in the runbook rather than a bug to wait out. Deleting the pod forces re-enumeration.

6. Stop DCGM from overwriting the real Xid.

The shipped etc/default-counters.csv contains exactly this:

DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
# DCGM_EXP_XID_ERRORS_COUNT, gauge, XID errors observed during the configured time window.
# DCGM_EXP_XID_ERRORS_TOTAL, counter, Total XID errors observed since exporter start.

The enabled metric is a gauge whose value is the last Xid seen. Because 154 is emitted in response to the fault, it is always last, so the panel reads 154 and the 79 that caused it is gone before anyone opens Grafana. The two metrics that carry the Xid as a label ship commented out. Uncomment them in the metrics ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: dcgm-metrics
  namespace: gpu-operator
data:
  dcgm-metrics.csv: |
    DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
    DCGM_EXP_XID_ERRORS_COUNT, gauge, XID errors observed during the configured time window.
    DCGM_EXP_XID_ERRORS_TOTAL, counter, Total XID errors observed since exporter start.

kubectl -n gpu-operator rollout restart daemonset/nvidia-dcgm-exporter
kubectl -n gpu-operator rollout status daemonset/nvidia-dcgm-exporter

Then key alerts on the per-Xid series with its Xid label instead of on a single overwritten value.

7. Optional: automate the tier-to-action mapping.

NVIDIA's NVSentinel (Apache-2.0) runs cordon, drain and remediate on GPU nodes, sourcing faults from DCGM, syslog/journalctl and CSP maintenance events:

helm upgrade --install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.21.0 \
  --namespace nvsentinel --create-namespace \
  --set podMonitor.enabled=false \
  --wait

Its fault-remediation module maps recommended actions onto CRDs (RESTART_VM becomes RebootNode, COMPONENT_RESET becomes a GPUReset scoped by GPU_UUID) and filters out NONE and UNKNOWN. That is the same two-tier split as the hex value, which tells you the tiering is the intended interface. Hand triage is the manual version of it.

Verify it works

nvidia-smi -q | grep -i "GPU Recovery Action"
kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
kubectl run gpu-smoke --rm -it --restart=Never \
  --image=nvidia/cuda:12.8.0-base-ubuntu24.04 \
  --overrides='{"spec":{"nodeName":"'"$NODE"'","containers":[{"name":"gpu-smoke","image":"nvidia/cuda:12.8.0-base-ubuntu24.04","command":["nvidia-smi","-L"],"resources":{"limits":{"nvidia.com/gpu":1}}}]}}'

Three results have to line up: recovery action reads None, allocatable is back to the full device count, and the smoke pod lists every GPU by UUID. If the count is right but a specific UUID is missing from nvidia-smi -L, that device never came back and step 4 did not finish.

Common pitfalls

• Alerting on the string Xid 154. A transition to 0x0 (None) means recovery completed. Key the alert on the destination value, or on-call gets paged by good news at 3 a.m.
• Blackwell soft reboots that leave the fault in place. In the dbirks/home-k8s reproduction, both a soft reboot and nvidia-smi -r failed on a GB202 after Xid 79 plus 154, and only a full power cycle recovered the card. The reported mechanism is that the driver does not issue a true Secondary Bus Reset, so the WPR2 secure region stays dirty. Reducing gpu_memory_utilization did not prevent recurrence there either. If 0x2 survives a reboot, stop retrying the reboot.
• Misreading the CDI error that follows. failed to create NVIDIA Container Runtime: failed to generate CDI spec for mode "auto": failed to get device handle from UUID: Unknown Error is the container runtime enumerating a GPU the driver can no longer address. It is downstream of the Xid, and chasing it as runtime configuration wastes an hour, the same trap as the conflicting SELinux label errors in K8s 1.37 that look like a mount bug and are not.
• Escalating Xid 13 and 31 as node faults. Both sit on the plugin's ignore list by design and map to RESTART_APP in NVIDIA's catalog. They will never move allocatable, and rebooting the node for them costs you a node and fixes nothing.
• The counterargument on DP_DISABLE_HEALTHCHECKS. Setting it to xids stops capacity flapping and keeps a node schedulable through transient Xids. On a fleet with per-job retry and short jobs, that is a defensible trade: you eat a few failed jobs instead of losing a whole node's scheduling for every blip. The price is that the scheduler will place work on a GPU sitting one fault away from 0x2, and you lose the XidCriticalError log line that step 3 reads. I keep health checks on and pay the flapping, because a silent bad GPU costs more debugging time than a re-queued job.

FAQ

Does Xid 154 mean my GPU is failing? By itself it means nothing about hardware health. It records a change in the recommended recovery action. Read the preceding Xid for the fault and the hex value for the action.

Can I just always reboot the node and skip the decision? You can, and some teams do. It costs you every other pod on that machine, and for 0x1 faults a targeted nvidia-smi -r returns the GPU in under a minute.

Why does allocatable stay low after a successful reset? The device plugin caches the unhealthy verdict and does not re-query. Delete its pod on that node (step 5).

Which driver version do I need for the recovery action field? R570 or newer exposes it through NVML. Older drivers emit neither Xid 154 nor the GPU Recovery Action field in nvidia-smi -q, so you diagnose purely from dmesg.

Wrap-up

Read the Xid before 154 for the fault, read the hex after to for the action, confirm the live tier from the driver rather than the log, and restart the device plugin because Kubernetes will not give the GPU back on its own. The single change with the longest tail is step 6: without the DCGM_EXP_ metrics, every future incident arrives in Prometheus as 154 with the cause already overwritten.

Next: wire DCGM_EXP_XID_ERRORS_TOTAL into an alert keyed on the Xid label, then decide whether NVSentinel's cordon-drain-remediate loop earns its place before the fleet grows past the size where a human still notices a missing GPU.