An H100 node that refuses every MIG change after a driver upgrade comes back under MIG control without a reboot, if you run six operations in one specific order. The fix that every Xid 119 thread on the internet recommends is not one of them, and on an operator-deployed driver it silently does nothing at all.

The symptom: you moved an H100 pool from driver 595.91.07 to 615.71.09, the driver pod reports Running, every label looks healthy, and the next MIG change never lands. You relabel the node and nvidia.com/mig.config.state stops short of success. Inside the driver container, destroying a GPU instance fails with NVML ERROR_UNKNOWN, and dmesg carries an Xid 119.

That exact pairing is documented in the GPU Operator 26.7.1 release notes: "On nodes with NVIDIA H100 GPUs, upgrading the driver from 595.91.07 to 615.71.09 without rebooting or resetting the GPUs can prevent subsequent MIG reconfiguration. Deleting a MIG GPU instance fails with NVML ERROR_UNKNOWN, and the driver logs Xid 119." 26.7.1 only just added support for 615.71.09, and 26.7.0 shipped 595.91.07 as its component version, so most search results you will find for this error predate the upgrade edge that produces it.

Prerequisites

  • GPU Operator 26.7.1 (MIG Manager v0.15.1, Driver Manager v0.12.1), with MIG Manager enabled and an nvidia.com/mig.config label already in use. MIG gives you hardware-partitioned slices, with the isolation limits I measured against time-slicing still worth reading before you pick a profile.
  • H100 nodes currently on 595.91.07, moving to 615.71.09. The condition is documented for that pair on H100; see the named gap under Common pitfalls before assuming it generalises.
  • kubectl with exec into the gpu-operator namespace, plus permission to label and drain nodes.
  • A maintenance slot per node. MIG Manager expects no user workloads on the GPUs it reconfigures, per the GPU Operator MIG docs. If your GPU pods carry nvidia.com/gpu.deploy.client=true, the operator restarts them for you during a driver upgrade or MIG change (added in 26.7.0).
  • Export the node once: export NODE=<node-name>.

Step-by-step

1. List the candidate nodes, then distrust the labels

kubectl get nodes -l nvidia.com/mig.config \
  -o custom-columns='NODE:.metadata.name,\
PRODUCT:.metadata.labels.nvidia\.com/gpu\.product,\
DRIVER:.metadata.labels.nvidia\.com/cuda\.driver\.major,\
MIG:.metadata.labels.nvidia\.com/mig\.config,\
STATE:.metadata.labels.nvidia\.com/mig\.config\.state'

You get the H100 nodes under MIG management and the state each one reports. The driver label records what the installer wrote into the node object. It says nothing about whether the GPUs still hold a stale context from before the swap, and that context is the failure condition.

On an unrelated patch day last month I checked a Brotli RCE by running the behaviour instead of reading the release string, because distro and vendored builds make a version number a liar in both directions. The probe disagreed with the version table and the probe was right. Same discipline here: the teardown attempt in step 2 is the only measurement that tells you which nodes are affected.

2. Probe one GPU instead of inferring from the version

DRV=$(kubectl -n gpu-operator get pod --field-selector spec.nodeName=$NODE \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep nvidia-driver)
kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi mig -lgi
kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi mig -dci -i 0
kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi mig -dgi -i 0

Compute instances come down before GPU instances. Reversing those two lines returns In use by another client, a different failure with a different fix, and reversing them is the most common way people misdiagnose this. A node in the documented state fails on the -dgi line with NVML ERROR_UNKNOWN.

3. Confirm the Xid, and read the whole buffer

kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- \
  bash -c "dmesg | grep -E 'NVRM: Xid.*: 119'"

NVIDIA's Xid catalog lists 119 as GSP_RPC_TIMEOUT, "GSP RPC Timeout", applicable to A100, H100, B100 and GB200, with an immediate-action bucket of RESET_GPU and an investigatory bucket of INVESTIGATE_SW. The catalog's note reads: "These events (119/120) may be logged when an error occurs in code running on the GSP core of the GPU and/or a timeout occurs while waiting for the GSP core of the GPU to respond to an RPC message." Grep the full buffer rather than the tail, because a DCGM-driven alert path can surface a later, louder Xid and bury this one.

4. Disable MIG before the driver moves

kubectl label node $NODE nvidia.com/mig.config=all-disabled --overwrite
kubectl get node $NODE \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'

Wait for success. The release notes are blunt about the branch here: "If disabling MIG fails, reboot the node instead of continuing with these steps." A node that cannot disable MIG is already stale, and no further label work will move it.

5. Pause the five operands that hold GPU handles

for k in device-plugin gpu-feature-discovery dcgm dcgm-exporter mig-manager; do
  kubectl label node $NODE nvidia.com/gpu.deploy.$k=false --overwrite
done

Each of these keeps an NVML handle open, and the reset in step 7 fails while any one of them is attached. People skip this step because dcgm-exporter and gpu-feature-discovery read like optional telemetry.

6. Move the driver version

# Helm-managed ClusterPolicy
kubectl patch clusterpolicy cluster-policy --type merge \
  -p '{"spec":{"driver":{"version":"615.71.09"}}}'

# or, if you manage drivers with NVIDIADriver CRs (26.7.0 added upgradePolicy there)
kubectl patch nvidiadriver <name> --type merge \
  -p '{"spec":{"version":"615.71.09"}}'

Wait for the driver pod on that node to report Running, as the workaround specifies. 26.7.0's driver-upgrade controller compares the configuration digest and may restart the driver pod in place, cordoning the node without draining it, so a Running pod is not evidence that the GPUs were reset.

7. Reset the GPUs from inside the driver container

kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi -r
kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi -L

nvidia-smi -r is the non-reboot half of the documented workaround, and a GPU reset is the action NVIDIA's own catalog assigns to Xid 119. The -L call confirms the devices came back before you hand the node to anything else.

8. Resume the operands, then restore the profile

for k in device-plugin gpu-feature-discovery dcgm dcgm-exporter mig-manager; do
  kubectl label node $NODE nvidia.com/gpu.deploy.$k=true --overwrite
done
kubectl label node $NODE nvidia.com/mig.config=all-1g.10gb --overwrite

Operands first, profile second. MIG Manager needs the device plugin and GFD running again before the new partitioning gets advertised into node capacity.

mig.config.state stuck, not successProbe: nvidia-smi mig -dci then -dgiSet mig.config=all-disabledWrong teardown order, retry -dci firstPause 5 gpu.deploy.* operandsReboot the node, stop herePatch driver.version to 615.71.09nvidia-smi -r, then nvidia-smi -LResume operands, relabel mig.configstate=success, MIG devices in node capacity"NVML ERROR_UNKNOWN + Xid 119""In use by another client""state=success""disable fails"

Verify it works

kubectl get node $NODE \
  -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}'
kubectl -n gpu-operator exec $DRV -c nvidia-driver-ctr -- nvidia-smi -L
kubectl get node $NODE -o jsonpath='{.status.capacity}' | tr ',' '\n' | grep nvidia

Three conditions have to hold together. The state label reads success. nvidia-smi -L lists MIG 1g.10gb Device entries under the physical GPU. Node capacity advertises the slices, as nvidia.com/gpu under the single strategy or nvidia.com/mig-1g.10gb under mixed, which is also the number your chargeback records will bill against. Then re-run the step 2 probe on one instance, because a correct capacity count coexists happily with broken teardown, and check that no new 119 has landed in dmesg since the reset.

Common pitfalls

NVreg_EnableGpuFirmware=0 is the wrong answer here. It is the top-voted fix in every Xid 119 thread, and the GPU Operator's -open driver images cannot honour it. NVIDIA's known-issue page for bug 6353226 states that GSP firmware cannot be disabled when the open GPU kernel modules are installed, and open-gpu-kernel-modules issue #1490, still open and reporting Xid 119/175/120 on both 595.91.07-open and 615.71.09-open, shows /proc/driver/nvidia/params printing EnableGpuFirmware: 0 while the module ignores it. The parameter reads back as applied and changes no behaviour, so you lose an afternoon confirming a setting that was never in force.

ERROR_LIBRARY_NOT_FOUND is a different bug. MIG Manager reporting that string traces to operands starting before the driver libraries exist after a driver pod restart, filed as gpu-operator issue #2874 and fixed in 26.7.1 so that, per the release notes, "GPU operands now wait for driver validation to complete." If that is your string, upgrade the operator and stop; this runbook does not apply.

Operands paused forever. Before MIG Manager v0.15.1, the manager could leave operands paused indefinitely while it waited for client pods to stop. The 26.7.1 fix restores the operand labels and reports the MIG configuration state as failed. On 26.7.1 a failed label is therefore a real signal, and on older builds the absence of one tells you nothing.

You cannot reboot a managed node. On EKS, GKE or AKS the documented reboot means replacing the instance: kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data, then delete the node through the cloud provider and let the node group recreate it. Replacement also pulls whatever node image the group now points at, and Kubernetes 1.37 dropped 18 kubelet flags that will stop the new node joining if your bootstrap config still sets them. The in-place reset path above exists so that you do not have to take either risk.

The gap I cannot close. The release notes scope the issue to H100 and to exactly 595.91.07 into 615.71.09. They do not say whether other source versions into 615.71.09 behave the same way, nor whether B100 or GB200 nodes are affected, while NVIDIA's Xid catalog marks 119 as applicable to A100, H100, B100 and GB200 alike. Nothing I have access to settles it. What settles it on a given node is one command, nvidia-smi mig -dci -i 0 followed by nvidia-smi mig -dgi -i 0 inside the driver container, so run that pair once per architecture and per source version before a fleet-wide rollout rather than extrapolating from this page.

Wrap-up

You now have an affected node back under MIG control with no reboot, and a sequence that holds for the other nodes in the pool: disable MIG, pause the five operands, bump the driver, reset, resume, relabel. Wire the step 2 teardown probe into whatever runs after your driver rollout, so the next upgrade reports its own damage instead of waiting for a partition request to fail days later.

What I read