Fix "a mirror pod may not reference secrets" in K8s 1.37

Kubernetes v1.37 goes GA tomorrow, Wednesday 26 August 2026, per the release team's schedule. Under the ipvs deprecation and the containerd 2.0 floor sits a two-line removal that closes a four-year-old reporting hole: the PreventStaticPodAPIReferences feature gate is gone, merged as PR #140226 on 6 July 2026. Short version: a static pod that references a Secret, ConfigMap, ServiceAccount, or PVC is now refused by the kubelet with no opt-out, and any node still pinning that gate to false will fail to start on 1.37. Two greps on one node per pool tell you whether you are affected.

What actually happened when a static pod referenced a Secret?

The rule itself is documented and boring. The Static Pods docs state that "the spec of a static Pod cannot refer to other API objects, such as ServiceAccount, ConfigMap, or Secret." What almost nobody has looked at is the behavior when you ignored it, because the failure was split across two subsystems that never compared notes. The container started. The Pod object did not exist. Both were true at once, and only one of them was visible from kubectl.

Mike Spreitzer filed issue #103587 on 8 July 2021 with the log lines that prove it. The kubelet writes "Creating a mirror pod for static pod" pod="default/test5-init1", then immediately "Failed creating a mirror pod for" err="pods \"test5-init1\" is forbidden: a mirror pod may not reference service accounts". The workload keeps serving traffic the entire time. That asymmetry survived until v1.34 put a fix behind a gate, and v1.37 welds the gate shut.

The 403 comes from the NodeRestriction admission plugin. In current master, plugin/pkg/admission/noderestriction/admission.go formats the message generically as "node %q can not create pods that reference %s", alongside "error checking mirror pod for API references: %w". That %s is filled by podutil.HasAPIObjectReference() in pkg/api/pod/util.go, which walks the spec and returns the plural resource name it found: serviceaccounts, secrets, configmaps, resourceclaims, persistentvolumeclaims, plus qualified variants like secrets (via azureFile volumes), csidrivers (via CSI volumes), and persistentvolumeclaims (via ephemeral volumes).

Two things follow for operators. The wording changed between releases, so older nodes print the 2021-era phrasing (a mirror pod may not reference service accounts) and newer ones print the generic form. Grep for mirror pod and read the suffix to learn which resource tripped it. And that helper returns an error on any spec field it does not recognise, so the check fails closed as new volume types land. That is the design choice that made removing the gate safe three releases after it shipped.

Why an invisible pod is worse than a failed one

A pod that runs without a Pod object is missing from the entire governance stack. Kyverno and Gatekeeper are admission webhooks on the API server; static pods bypass admission by construction, and the mirror pod is the one API artifact that would have exposed them. CIS benchmark scanners, image inventories built from kubectl get pods -A -o json, runtime-to-API reconciliation in a SIEM, cost allocation, and every "what is running in my cluster" dashboard read the same source of truth, and that source of truth has a hole in it.

I care about this more than the raw severity suggests because of what it does to inventory. If you build an SBOM per running image from an API-server listing, the container you most want to inspect (a hand-edited control-plane component pulling from a private registry) is precisely the one missing from the input list. Add an imagePullSecrets block to a static pod manifest and the container gets harder to see while staying just as easy to run. That is a strange property for a security boundary, and it is why this landed as a removal rather than a docs clarification.

The population at risk is narrow. Vanilla kubeadm control-plane manifests use hostPath volumes only, so a stock cluster upgrades clean. The clusters that break are the ones where someone edited /etc/kubernetes/manifests by hand: private-registry pulls that added imagePullSecrets to kube-apiserver.yaml, air-gapped installs, and vendor node agents shipped as static pods with a projected serviceAccountToken volume for telemetry. If you have already moved workloads to bound ServiceAccount tokens, that projected volume is exactly the pattern you were encouraged to adopt, which is how well-meaning teams ended up here.

Three causes of "it runs but kubectl cannot see it"

The symptom is one line in a ticket. It has at least three unrelated causes, and the GitHub threads conflate all of them.

1. An API reference in the manifest. The kubelet log shows a create attempt followed by a 403 naming a resource. journalctl -u kubelet | grep -i "mirror pod" is the whole diagnosis.
2. A stale mirror pod after a hash mismatch. The Pod object exists but describes an older manifest. Compare kubectl get pod <name>-<node> -o jsonpath='{.metadata.annotations.kubernetes\.io/config\.hash}' against the value after a fresh kubelet restart. The kubelet deletes and recreates on divergence.
3. The kubelet cannot reach the API server, or the Node object is missing. Then no mirror pods exist for that node at all, while every static pod still runs. Count them:

kubectl get pods -A -o json | jq '[.items[]
  | select(.metadata.annotations["kubernetes.io/config.mirror"])
  | select(.spec.nodeName=="'$NODE'")] | length'

Zero means cause 3. A partial count means cause 1.

The check that separates all three without guessing is a diff between runtime truth and API truth, run on the node:

crictl pods -o json | jq -r '.items[].metadata.name' | sort > /tmp/runtime.txt
kubectl get pods -A --field-selector spec.nodeName=$NODE \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort > /tmp/api.txt
comm -23 /tmp/runtime.txt /tmp/api.txt

Anything in the first list and absent from the second is your invisible set. It takes five seconds per node class, and it is the only inventory that does not inherit the bug. Run it on 1.33 and older too, where nothing else will tell you.

The kubelet config line that stops a node from registering

The failure that actually pages you at 03:00 is smaller than the security story. A node whose KubeletConfiguration still carries PreventStaticPodAPIReferences: false, pinned during a v1.34 or v1.35 upgrade to defer this work, meets a removed gate on v1.37. The kubelet exits at startup with failed to set feature gates from initial flags-based config: unrecognized feature gate, and the node never registers. k3s users saw the identical shape when KubeletCredentialProviders was removed in 1.28.

This is the same class of upgrade break as the unrecognized format int32 surprise in 1.34: a change that is trivially fixable if you find it in a canary node, and miserable if you find it after draining half the control plane. Do the node-side prep first, the way you would for enabling user namespaces with hostUsers false, and keep the config edit in a separate change from the version bump.

The fix in circulation is the wrong artifact

The workaround people reach for is to keep imagePullSecrets in the manifest and disable the gate. That credential never worked for a static pod in the first place. Registry auth for control-plane images comes from the runtime: /etc/containerd/certs.d/<registry>/hosts.toml, the registry.configs.<host>.auth section of config.toml, or a kubelet credential provider configured with --image-credential-provider-config. Clusters that "proved" imagePullSecrets worked were pulling from a warm image cache or an already-authenticated containerd, and the proof evaporates on a rebuilt node.

Here is my opinion, and it is arguable: I would rather see node-level config and credentials in hostPath files with 0600 ownership, owned by whatever config management already owns the node, than reachable through the API server. Yes, that means the value is not rotated by a Secret controller and not visible in kubectl describe. A control-plane component that depends on the API server to fetch the credential it needs to start is a bootstrap loop waiting for a bad day, and the same reasoning applies to any node agent you expect to keep working while the API server is down.

What to run before you upgrade to 1.37

In this order, on one node per pool, this week:

1. Grep the kubelet config for the dead gate. grep -n PreventStaticPodAPIReferences /var/lib/kubelet/config.yaml. A hit means that node fails to start on 1.37. Delete the line, restart the kubelet on the current version, confirm the node stays Ready, then upgrade. This is the only step that causes an outage if skipped.
2. Grep the manifests for references. Read staticPodPath from the config first, because it is not always /etc/kubernetes/manifests:

SPP=$(awk '/staticPodPath/{print $2}' /var/lib/kubelet/config.yaml)
grep -REl 'secretRef|configMapRef|imagePullSecrets|serviceAccountName|persistentVolumeClaim|projected' "$SPP"

Every file listed is a node that will lose its static pod on 1.37.

1. Run the invisible-pod diff from the section above on each node class and keep the output. If it is non-empty on a pre-1.34 node, you have been running unreported containers, and your compliance evidence has been under-reporting since 8 July 2021 when the behavior was first written down.
2. Move registry credentials into containerd (hosts.toml or a credential provider) and node config into hostPath files, then delete the offending manifest lines. Verify with a cold pull: crictl rmi <image> && crictl pull <image> on a node with no cached layer. A pull that succeeds only because the layer was already local is the failure mode that hides here.
3. Add step 3 to your node readiness check permanently. Treat a non-empty diff as a failed check, the same way you would treat a default-deny egress policy that never got applied.

What I read while writing this: the v1.37 sneak peek for the release framing, PR #140226 for the removal itself, issue #103587 for the 2021 logs, and the Static Pods documentation for the rule that was always there.