GPU Chargeback on Shared GPUs: 12 Tips, 1 Unbillable Metric

You turned on time-slicing to lift GPU utilization, four teams now share an H100, and finance wants a per-team number. The dashboard already plots DCGM_FI_PROF_GR_ENGINE_ACTIVE with a pod label beside it, so the report looks like a morning's work. It is not. That series is a device-wide figure stamped with whichever pod the exporter joined it to, and three pods on one GPU can each read 100%.

Short version: DCGM exposes two metric families and only one of them can be attributed to a tenant. The DCGM_FI_PROF_ fields are backed by NVML GPM calls that accept no process ID, so per-pod profiling numbers are impossible by construction. The DCGM_FI_DEV_ fields gained per-process values in dcgm-exporter 4.5.3 (May 2026), behind a flag, with a privileged pod as the price. The twelve tips below separate the number you can defend in a dispute from the number you cannot.

NVIDIA's own GPU Operator documentation still states that per-pod attribution under sharing is not available, which is why most advice on this topic stops at "use showback." The field family that changed, and the admission cost of switching it on, are what the rest of this covers. If you have not yet confirmed which pools are shared and what isolation they lack, the eleven checks on GPU sharing isolation come before any of this.

Twelve tips, in the order a cost review asks for them

1. Grep your rules and dashboards for DCGM_FI_PROF_ joined to a pod label, then delete those panels from the billing path. Maintainer nccurry closed dcgm-exporter issue #307 on 6 April 2026 with the reason that per-pod profiling metrics with time-slicing are not possible at the DCGM or NVML level, because the GPM functions behind DCGM_FI_PROF_* "operate at device/MIG-instance level only and don't accept a process ID." The sibling issue #587 shows the symptom in the raw: three pods, one GPU, DCGM_FI_PROF_GR_ENGINE_ACTIVE reading 0.714 for each of them.

    grep -rn 'DCGM_FI_PROF_' dashboards/ prometheus/rules/

1. Move the invoice onto the DCGM_FI_DEV_ pair and check the exporter version under it. PR #594, merged 24 March 2026, added per-process DCGM_FI_DEV_GPU_UTIL (SM utilization) and DCGM_FI_DEV_FB_USED (framebuffer, MiB) via nvmlDeviceGetProcessUtilization and nvmlDeviceGetComputeRunningProcesses. It shipped in release 4.5.3-4.8.2 on 7 May 2026. Anything older emits device values under pod labels, which is the failure in tip 1 wearing a different field name.

    kubectl -n gpu-operator get ds nvidia-dcgm-exporter \
      -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

1. Price the admission exception before you promise finance a date. Per-process attribution wants KUBERNETES_VIRTUAL_GPUS=true, DCGM_EXPORTER_KUBERNETES=true, DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID=true, a correct DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE (device-name on GKE, uid on EKS), plus hostPID: true and a privileged security context, because the PID-to-pod mapping walks process cgroups. A restricted Pod Security Standard on the monitoring namespace rejects that DaemonSet, and the replacement policy engines do not all offer an equivalent escape hatch: podSecurity rules have no CEL equivalent after the Kyverno ClusterPolicy removal. Write the exception and its named owner into the change, not into a Slack thread.
2. Probe the exporter's output rather than trusting its tag. Vendor rebuilds and operator-managed images make a version string a poor witness. On a Node 20 box I took a Brotli decoder's own error code over the release number and found the runtime unpatched where the version table said otherwise, the same habit I argue for in the version string lies. Scrape the endpoint and compare two pods sharing one gpu= index: identical values mean you are still reading device scope.

    POD=$(kubectl -n gpu-operator get pod -l app=nvidia-dcgm-exporter \
      -o jsonpath='{.items[0].metadata.name}')
    kubectl -n gpu-operator exec "$POD" -- curl -s localhost:9400/metrics \
      | awk '/^DCGM_FI_DEV_FB_USED/ && /pod=/' | sort -u

1. On MIG, bill memory-time and stop hunting for per-process SM time. PR #594 is explicit that for MIG "Only memory metrics are available per-process (NVML limitation - SM utilization not available for MIG)." A MIG rate card therefore prices the profile and the hours, GB-hours of a 1g.10gb slice, and treats compute as bundled into that price. nvidia-smi -L prints the instance UUIDs your ledger keys on.
2. Rule out nvidia-smi accounting as an invoice source, in writing. NVML's accounting statistics documentation states that "Accounting stats are kept in a circular buffer, newly created processes overwrite information about old processes," that the mode "is not persistent and will default to disabled after driver unloads," and that on MIG-enabled GPUs it "would be set to DISABLED and changing it is not supported." A monthly bill cannot rest on a buffer that a driver reload empties.

    nvidia-smi -q | grep -i -A1 'Accounting Mode'
    nvidia-smi --query-accounted-apps=pid,gpu_util,max_memory_usage --format=csv

1. Alert on the ledger inventing GPU time. PR #594 notes that device-level series are "always emitted alongside per-pod metrics," which hands you a free denominator. Any result from the expression below means attributed usage exceeds the hardware you own. The label set depends on your enrichment config, so confirm pod actually exists on the series before you read silence as agreement.

    sum by (Hostname, gpu) (DCGM_FI_DEV_GPU_UTIL{pod!=""})
      > on (Hostname, gpu)
    sum by (Hostname, gpu) (DCGM_FI_DEV_GPU_UTIL{pod=""})

1. Fix the requests side, which double-bills before utilization enters the picture. The OpenCost specification defines workload cost as max(request, usage) for allocated resources, and its measurement table reads "the greater of requested resources and used resources" for the GPU row. With replicas: 4, four pods each requesting nvidia.com/gpu: 1 sum to four GPUs on a node that owns one. Your cost engine is behaving correctly on an integer the device plugin minted.
2. Know what renameByDefault gives you and what it takes away. Setting it to true advertises the shared resource as nvidia.com/gpu.shared, so shared and exclusive requests can carry different prices. The same GPU Operator docs state that in this mode the nvidia.com/gpu.product label is left unmodified, which strips the -SHARED suffix a node-pool rate card may already key on. Change both ends in one commit.

    sharing:
      timeSlicing:
        renameByDefault: true
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4

1. One replica count per node pool, one row on the rate card. The replica count doubles as your oversubscription ratio and your tenant discount, and NVIDIA is blunt that there is no memory or fault isolation between replicas. Mixing counts inside a pool leaves the per-replica price unknowable at admission time, when the scheduler has already picked the node.

    kubectl get nodes -L nvidia.com/gpu.replicas,nvidia.com/gpu.product

1. If you are migrating to DRA, the join key moved with you. Release 4.8.4, dated 18 September 2026 in the dcgm-exporter releases, added attribution of DRA MIG metrics to pods when the MIG device is visible to DCGM at startup, refreshes on ResourceSlice changes, and a GPU_CI_ID label for compute-instance-scoped fields. An older exporter drops the pod label when slices churn, and a gap in a usage ledger bills as free GPU time. Driver visibility is the recurring fault in these pools, which is where NVIDIA-SMI has failed in a DRA pod starts.
2. Charge reserved replica-seconds and keep utilization as a showback column. PR #594 records that pods with no active GPU processes show 0 for both metrics, so a pure usage model bills a squatter nothing while its replica stays unavailable to every other team. Take the allocation side from the kubelet pod-resources socket, which offers no server reflection, so pass the proto from the Kubernetes tree explicitly.

    grpcurl -plaintext -proto api.proto \
      -unix /var/lib/kubelet/pod-resources/kubelet.sock \
      v1.PodResourcesLister/List

Wrap-up

Run the reconciliation in tip 7 as a monthly alert: attributed GPU-seconds against owned GPU-seconds, per node and per gpu index, never as a spreadsheet somebody remembers to open. Pair it with a footnote on every chargeback statement covering time-sliced pods, naming the compute component that no NVML call can measure per process. A tenant who discovers that limitation during a billing dispute stops trusting the invoice, including the parts of it you can prove.

Standing this up is four concrete pieces of work: splitting pools by replica count, carrying a privileged hostPID exporter through your admission policy, writing the reconciliation rule, and turning the output into a rate card a tenant will sign. The advisory packages cover that review against your own cluster and its node pools.