Fix kubelet "not run on a host using cgroup v1" on v2

The kubelet exits with failed to validate kubelet configuration, error: kubelet is configured to not run on a host using cgroup v1, you SSH to the node, run mount | grep cgroup2, and get a hit. The mount is real. It is mounted somewhere the kubelet never looks.

TL;DR: The check behind that error is a single statfs on /sys/fs/cgroup compared against CGROUP2_SUPER_MAGIC. Run stat -fc %T /sys/fs/cgroup on the failing node: cgroup2fs passes, tmpfs fails, and you have reproduced the kubelet's entire opinion of the host in one command. Two things read as v1 on a machine that looks unified: a systemd hybrid layout, and a GRUB edit that the bootloader never applied.

What does the kubelet actually check?

IsCgroup2UnifiedMode() statfs's one path and compares f_type to one magic number. The Kubernetes cgroups documentation names the same test in human form: stat -fc %T /sys/fs/cgroup/ returns cgroup2fs on v2 and tmpfs on v1.

Every other command an operator reaches for during triage answers a different question. mount | grep cgroup2 reports whether a cgroup2 filesystem exists anywhere in the namespace. cat /proc/filesystems reports kernel support. podman info reports what podman decided for itself. All three will cheerfully confirm you "have cgroup v2" on a node the kubelet has already rejected, which is why this failure survives a full afternoon of triage.

Timing makes it worse. Kubernetes v1.37 went GA on 26 August 2026, and fleets that skipped 1.35 and 1.36 are meeting the v1.35 default flip for the first time this month, in the same upgrade window that removed the static-pod secretRef escape and turned SELinuxMount on by default. Three node-level behavior changes arriving together means the first symptom you see is rarely the one you are debugging.

Why does a node with cgroup2 mounted still fail?

Hybrid mode is the most common false v1. systemd's hybrid layout mounts cgroup2 at /sys/fs/cgroup/unified and leaves the v1 controllers on the root. The root stays tmpfs, the statfs returns v1, and grep cgroup2 /proc/mounts returns a hit at a path the kubelet does not inspect.

The upstream cgroups page documents only the fully unified path, systemd.unified_cgroup_hierarchy=1 in GRUB_CMDLINE_LINUX, and says nothing about the hybrid layout. Ask a coding assistant trained on that page why your node fails and it will send you back to a document that does not describe your machine.

The second false v1 is a GRUB edit that never took. Those docs end with sudo update-grub, a Debian-family wrapper. On RHEL-family hosts using Boot Loader Specification entries, the kernel command line lives in /boot/loader/entries/*.conf and grubenv, so editing /etc/default/grub and regenerating grub.cfg changes nothing the bootloader reads. The node reboots, the operator sees their edit in /etc/default/grub, and concludes the fix is applied. /etc/default/grub records intent. cat /proc/cmdline records state.

Third, and rarer, the mount namespace. kind nodes, Docker-in-Docker CI runners, and containerized kubelets see whatever /sys/fs/cgroup their own namespace was handed. The host can be fully unified while the kubelet's view is not, which is the one case where stat from an SSH session and stat from inside the kubelet disagree.

Why this takes out a generation of nodes at once

A kubelet that refuses to start does not give you a NotReady node. It gives you a node that never registers, with no pod evictions to watch and no partial service to degrade gracefully. Managed control planes mostly protect you here because the vendor maintains the image. Bare metal, RKE2 and K3s clusters, on-prem VMware estates, and hand-rolled AMIs carry a kernel command line that someone edited in 2021 and nobody has read since.

The failure arrives per-image, so it is uniform across every node built from the same golden AMI. A canary pool of three workers from that image looks clean or looks broken together, and the same batch behavior takes out a whole generation of workers in one rollout. Anyone who has staged a Cilium upgrade across a mixed node fleet already knows the shape: the risk unit is the image, not the node.

Find the cgroup v1 nodes before you upgrade anything

KEP-4569 added a fleet-wide signal for exactly this and almost nobody runs it. The kubelet emits kubelet_cgroup_version at startup with a value of 1 or 2, added explicitly so admins stop inspecting nodes by hand (KEP-4569, cgroup v1 maintenance mode):

kubectl get --raw /api/v1/nodes/<node>/proxy/metrics | grep kubelet_cgroup_version

The caveat is the reason to run it early: the metric comes from a running kubelet. After the upgrade, the node that failed validation emits nothing, so the signal disappears at precisely the moment you want it. Scrape it while the old kubelet is still alive, and treat every 1 as a node that will not come back.

What the two circulating workarounds actually cost

Kubernetes 1.35 made the SystemVerification preflight check error when it detects cgroup v1 alongside a kubelet at 1.35 or newer. That check is now coupled to something unrelated: issue #135875, opened 21 December 2025, reports that a containerized kubelet cannot answer kubelet --version, which fails the entire SystemVerification check for reasons that have nothing to do with cgroups. The workaround spreading from that thread is --ignore-preflight-errors=SystemVerification, which suppresses the cgroup detection along with everything else in that check. You would be disabling the one control that was going to warn you, to route around a version-string bug.

failCgroupV1: false is the honest escape hatch, and it has a price and an expiry. It pins the node off every cgroup v2-only feature: Memory QoS tiered protection via memory.min and memory.low, which landed in 1.36, swap support, and PSI metrics. KEP-5573 commits to removing cgroup v1 code from Kubernetes entirely, "no earlier than 1.38." On a three-release-a-year cadence, the hatch you open today closes in roughly eight months.

A platform team with a funded reimage program landing inside two releases has a real case for setting failCgroupV1: false fleet-wide, shipping the upgrade on schedule, and letting image rotation solve cgroups. That buys a clean upgrade with zero node churn and costs Memory QoS, swap, and PSI until the images land. It is the wrong trade if your node images are hand-maintained, because the reimage that was going to happen anyway is the one that never happens.

Two clocks that both say "cgroup"

The CRI cgroup driver lookup went GA in v1.34. containerd implements the RuntimeConfig RPC from v2.0.0, CRI-O from v1.28.0, and the kubelet exposes kubelet_cri_losing_support to flag nodes whose runtime is too old to answer. Threads merge this with the v1/v2 question constantly because both say "cgroup." They are independent: a node can be perfectly unified on cgroup v2 and still be on the wrong side of the runtime clock. Check both in the same pass so you reboot each node once. On hand-maintained node images that already carry driver-level pinning, a single coordinated reboot is worth real planning effort.

Beyond the two KEPs, the reading that matters is the upstream cgroup transition traffic in kubernetes/kubernetes, #135875 and #132925.

Do this before your next node reboot

1. Scrape kubelet_cgroup_version across every node today, before any 1.37 rollout touches a kubelet binary. Every 1 is a node that will fail to register after the upgrade.
2. On each flagged node, run stat -fc %T /sys/fs/cgroup. That is the same call the kubelet makes, so its answer is the only one that predicts the outcome.
3. If it returns tmpfs, check grep cgroup2 /proc/mounts. A hit under /sys/fs/cgroup/unified is the hybrid layout: set systemd.unified_cgroup_hierarchy=1 on the kernel command line and reboot.
4. Confirm the boot state with cat /proc/cmdline, never with /etc/default/grub. On RHEL-family hosts with BLS entries, regenerate /boot/loader/entries/*.conf and re-read /proc/cmdline before you call the node fixed.
5. If the node is a kind cluster, a DinD runner, or a containerized kubelet, run the stat from inside the kubelet's namespace. The host being unified tells you nothing about what the process sees.
6. Where you need failCgroupV1: false, file it as a dated exception with a named owner and a removal date inside the 1.38 window. Log what the node gives up: Memory QoS, swap, PSI.
7. Leave SystemVerification enabled. If #135875 is blocking a containerized kubelet, fix the version-string path rather than widening the suppression.
8. In the same maintenance window, pull kubelet_cri_losing_support and schedule containerd 2.0.0 or CRI-O 1.28.0 upgrades for whatever it flags.