Kubernetes GPU Sharing: 11 Checks, No Memory Isolation

Someone switched on GPU time-slicing to lift utilization off 18%, the finance slide improved, and now a customer questionnaire asks whether tenant A can read tenant B's inference traffic. The short answer is that a time-sliced replica provides no isolation, NVIDIA states this in its own GPU Operator documentation, and the node label telling you which of your pools are shared takes one kubectl command to read. These eleven checks produce the evidence an architecture review or a security addendum needs, in the order I run them.

On 22 October I pointed a defensive probe at my own server and read the Brotli decoder's error code instead of node -v. It returned errno=-9 on a Node 20 line that went end of life on 30 April 2026, so the box was unpatched and the release number had told me nothing, which I wrote up in the Brotli probe that contradicted the version string. Every check below works the same way: ask the device or the API server, never the values file you typed your intent into. If the device does not answer at all, you are debugging a different problem, the one behind NVIDIA-SMI failing inside a DRA pod.

The checks

1. Read nvidia.com/gpu.sharing-strategy across the estate before you answer anybody. GPU Feature Discovery writes it next to nvidia.com/gpu.replicas, nvidia.com/mig.capable and nvidia.com/mig.config. Capacity on its own misleads you: a node advertising nvidia.com/gpu: 8 is either eight physical devices or one device cut eight ways, and the scheduler handles both identically.

kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.sharing-strategy,\
nvidia.com/gpu.replicas,nvidia.com/mig.config

An exclusive pool returns none and 1. Any row with replicas above 1 is a pool you cannot call isolated in writing.

1. Quote NVIDIA's sentence in the review document rather than a vendor blog's paraphrase. The GPU Operator time-slicing page says it flatly: "Unlike Multi-Instance GPU (MIG), there is no memory or fault-isolation between replicas." A primary vendor statement settles the argument in a security review faster than any benchmark you could run, and it moves the discussion to what the business will spend on remediation.
2. Set failRequestsGreaterThanOne, because the default lets one pod stack slices of a single card. It ships as false, so a workload asking for two GPUs can be admitted onto two replicas of the same physical device and will run at roughly half the throughput its author assumed. Flipping it to true makes that request fail visibly with an UnexpectedAdmissionError, which is the behaviour you want from any gate you are relying on, for the reasons in 11 ways Kubernetes admission control fails open.

sharing:
  timeSlicing:
    failRequestsGreaterThanOne: true
    resources:
      - name: nvidia.com/gpu
        replicas: 4

1. Assume your time-slicing ConfigMap edit never took effect. NVIDIA documents that "the Operator does not monitor changes to a time-slicing config map", so a remediation ticket closed on kubectl apply can stay false for weeks until the device plugin restarts. Node capacity is the only answer that counts.

kubectl get node "$NODE" \
  -o jsonpath='{.status.capacity.nvidia\.com/gpu}{"\n"}'

1. Tell finance that per-tenant GPU chargeback on a time-sliced pool is invented. The same NVIDIA page states that DCGM-Exporter cannot associate metrics with containers when time-slicing is enabled, so DCGM_FI_DEV_FB_USED has no workload to attribute it to. An internal invoice built on that series is a number with no measurement behind it. Billed tenants belong on MIG instances or exclusive devices.
2. Treat uninitialized device memory as a live finding with a CVE attached. In csrc/quantization/gguf/gguf_kernel.cu, vLLM typed the element-count parameter as a 32-bit int. Once a GGUF weight tensor passed INT_MAX elements the count truncated, the kernel filled only part of an output buffer allocated by torch::empty, and the unwritten tail still held whatever the previous tenant left in VRAM. That is CVE-2026-53923 / GHSA-5jv2-g5wq-cmr4, affecting vLLM >= 0.5.5, < 0.24.0, fixed in 0.24.0 by PR #44971, filed as CWE-200 plus CWE-681. Only the GGUF path reaches it, which is what separates an exposure from a panic. Inventory first, then decide, the same sequence that makes RAG chunks carrying no ACL a scoped problem instead of an open-ended one.

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | grep -i vllm

1. Claim MIG as a memory boundary and refuse to claim it as a timing boundary. "Behind Bars" (Gu, Levine, Zhang, Sorensen and Guo, USENIX Security '26, pages 5373-5392) shows memory-barrier operations issued inside one MIG instance perturbing load timing in another, enough to infer a victim's kernel-launch pattern. In LLM serving, launch patterns follow the shape of the work. Hardware partitions the memory; they do not conceal that a neighbour is busy.
2. Alert on nvidia.com/mig.config.state, never on the config you asked for. A label reading nvidia.com/mig.config=all-1g.10gb records an intention. MIG Manager writes the state label only after it drains GPU pods and reconfigures the device, so a node halfway through that sequence is partitioned differently from your diagram.

kubectl get nodes -l nvidia.com/mig.capable=true \
  -o custom-columns=NODE:.metadata.name,\
WANT:.metadata.labels.nvidia\\.com/mig\\.config,\
STATE:.metadata.labels.nvidia\\.com/mig\\.config\\.state

1. Clearing VRAM is an administrative action somebody has to schedule. NVIDIA ships --clear-memory in gpu-admin-tools, whose own help text reads "Clear the contents of the GPU memory. Supported on Pascal+ GPUs." The flag exists because the behaviour is not automatic between workloads. Pick the point in your node lifecycle where it runs, using the same tool you already invoke for mode queries.

sudo python3 ./nvidia_gpu_tools.py --devices gpus --query-cc-mode

1. If a contract promises encrypted GPU memory, verify the mode and look for devtools. Confidential Computing has three states, off, on and devtools, and devtools keeps the feature set while lifting the blocks on profiling and debugging. Ready state also resets on reboot, so a node that passed an audit last month can be serving unprotected traffic right now.

sudo nvidia-smi conf-compute -f     # expect: CC status: ON
sudo nvidia-smi conf-compute -grs   # expect: Ready state: ready

1. Do not promote the hypervisor to a stronger boundary without reading the driver bulletin. NVIDIA's 30 September 2026 bulletin covered GPU display driver and vGPU software with 114 CVEs, including CVE-2026-47498, an out-of-bounds write in the vGPU Manager's GSP plugin that a guest VM user can reach through a crafted RPC message. Pull your installed branch, match it against the fixed build for that branch, and only then let the vGPU row in your tenancy matrix say "hypervisor-enforced".

nvidia-smi --query-gpu=driver_version --format=csv,noheader

Wrap-up

Build the tenancy matrix once: one row per GPU node pool, carrying the sharing mode, the memory boundary, the fault blast radius in pods, the metrics attribution, and the date a human last checked it. The habit that keeps it honest is regenerating the first three columns from kubectl on a schedule, so the document ages alongside the cluster instead of alongside the diagram. Teams usually find at that point that the utilization win and the isolation claim came out of the same budget, and that moving billed tenants onto MIG profiles or exclusive devices is a capacity purchase with a number attached. Having that argument internally costs less than having it in front of a customer's security reviewer.

What I read while writing this: NVIDIA's GPU sharing documentation for the time-slicing, MPS and MIG behaviour, GHSA-5jv2-g5wq-cmr4 for the vLLM GGUF buffer, the USENIX Security '26 paper for the MIG timing channel, gpu-admin-tools for the memory-clear and CC-mode flags, and Google Cloud's Confidential Computing verification guide for the three-state mode check.