kube-proxy IPVS to nftables: Fix 11 Traps in 1.40

Kubernetes 1.40 does two things to kube-proxy in the same release. KEP-5495 flips the KubeProxyIPVS feature gate to Default: false, so IPVS mode stops starting unless you override it, and KEP-5343 makes nftables the default proxy mode, so any cluster that never set mode: gets moved underneath it. If you run IPVS today on a cluster with LoadBalancers, a NodeLocal DNSCache, and an alert rule someone wrote in 2021, the upgrade picks your dataplane for you.

These eleven tips are the behaviour differences that show up in production rather than in the release notes, each with the command that finds it before the rollout does.

The 11 traps

1. Settle three inputs before you touch a ConfigMap, then run the decision per node pool. Kernel version, whether your CNI already replaces kube-proxy, and how many Services you run determine the path, and they often differ between pools. If you are on Cilium with kubeProxyReplacement enabled there is nothing to migrate, though the upgrade itself has its own set of traps before BGP drops.

The "bump the node image" branch is a separate project with its own failure modes, including kubelet refusing to start with "not run on a host using cgroup v1" on a newer base image. Budget it as one.

1. Price the escape hatch: KubeProxyIPVS=true buys you exactly three releases. KEP-5495 lays out the schedule: 1.35 logs a deprecation warning, 1.37 introduces the gate at Default: true, 1.40 flips it to false and kube-proxy exits with an error unless you set it back, 1.43 locks the gate and deletes pkg/proxy/ipvs, 1.46 removes the gate entirely. Setting it in 1.40 is a legitimate stall with a hard expiry on the 1.43 upgrade, and there is no further extension to negotiate for.
2. Write mode: explicitly this week, even if you plan to stay on iptables. KEP-5343 states the failure directly: a cluster on a kernel too old for nftables that upgrades to 1.40 "without having explicitly set mode: iptables in their config" will start kube-proxy in nftables mode and fail. One line of config removes an entire class of upgrade-day incident, which is cheap next to the ones you find out about from a pager.

# kube-system/kube-proxy ConfigMap, config.conf
mode: "nftables"   # or "iptables", never blank

1. Adding --proxy-mode=nftables to the DaemonSet args does nothing. When kube-proxy starts with --config, command-line flags are ignored (kubernetes/kubernetes #98302), and kubeadm's DaemonSet runs --config=/var/lib/kube-proxy/config.conf. Check which one you have, then confirm the mode actually changed from the logs rather than trusting the rollout. Upgrade changes that silently no-op are a recurring shape in this ecosystem, in the same family as unrecognized format int32 in 1.34.

kubectl -n kube-system get ds kube-proxy \
  -o jsonpath='{.spec.template.spec.containers[0].command}'
kubectl -n kube-system logs ds/kube-proxy | grep -i 'proxier\|nftables'

1. Check the kernel on every node before you edit anything, because the ConfigMap is one object for the whole cluster. nftables mode needs kernel 5.13+ and nft 1.0.1+, and kube-proxy refuses to start below that. Sorting the Node objects by kernel version gives you the fleet answer in one command.

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion | sort -k2

The blind spot: that string is what kubelet saw when it started, so a node that later rebooted into a rollback GRUB entry still reports the newer kernel. Spot-check those with uname -r on the host. nftables.skipKernelVersionCheck is a development option that skips the check without shipping a newer nft, so it papers over the problem instead of solving it.

1. NodePort quietly stops answering on every node address except the primary. iptables and IPVS default nodePortAddresses to all local IPs; nftables defaults to primary, meaning the node's primary IPv4 and IPv6 per the Node object (PR #123105). A hardware load balancer pointed at a secondary NIC starts failing health checks the moment the DaemonSet rolls, and the Service looks healthy from inside the cluster the entire time. Keep the old behaviour deliberately, or list the CIDRs you actually front:

nodePortAddresses: ["0.0.0.0/0", "::/0"]

1. 127.0.0.1:<nodePort> is gone and will not come back as a default. KEP-3866 dropped loopback NodePort deliberately to avoid the route_localnet sysctl, which is the CVE-2020-8558 surface. KEP-6032 adds it back as an opt-in userspace TCP proxy behind KubeProxyNFTablesLocalhostNodePorts: TCP only, rejecting UDP and SCTP, and only when loopback appears in nodePortAddresses. Find the callers before the rollout by grepping host-network manifests, systemd units, and monitoring configs for 127.0.0.1:3[0-2][0-9][0-9][0-9]. The same inventory is worth having anyway when you get around to default-deny egress.
2. Expect intermittent "connection reset by peer" on long-lived connections, and answer it with tcpBeLiberal. iptables mode installs a DROP rule for packets that conntrack marks INVALID, and PR #120412 made that rule conditional so kube-proxy skips it when --conntrack-tcp-be-liberal is set. nftables mode never installs the rule at all, which means the resets described in issue #117924 can resurface on connections that idle through a conntrack window. Watch the per-CPU counters on a canary node first, then set the option:

conntrack -S | tr ' ' '\n' | grep invalid

conntrack:
  tcpBeLiberal: true

1. Your kube-proxy failure alert stops firing without ever going red. kubeproxy_sync_proxy_rules_iptables_restore_failures_total is exported in iptables and IPVS modes only; nftables mode exports kubeproxy_sync_proxy_rules_nftables_sync_failures_total under a different name. A counter alert on a metric that stopped existing stays silent forever, which is worse than no alert because the dashboard still looks green. Rewrite the rule ahead of the rollout and pair it with an absence check so a future rename pages you:

absent(kubeproxy_sync_proxy_rules_nftables_sync_failures_total) == 1

Metrics are on :10249/metrics, health on :10256/healthz.

1. Scale-test at your real Service count, because nftables has a reported cliff. kubernetes/kubernetes #135639, opened 2025-12-06 against 1.32 to 1.34 on Amazon Linux 2 and 2023 and Bottlerocket, reports sync times of 13m15.34210662s at 1000+ Services alongside kernel soft lockups (watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [nft:222752]). Check the current state of that issue against your exact patch version, then measure rather than trust either side of the argument: record the p99 of kubeproxy_sync_proxy_rules_duration_seconds on IPVS as your baseline and compare it on the canary pool under the same Service count.
2. Leave your NodeLocal DNSCache configuration exactly as it is. The Kubernetes docs give two recipes: under IPVS the node-local-dns pod listens only on the link-local address and kubelet's --cluster-dns points there, because the CoreDNS ClusterIP is already claimed by the kube-ipvs0 dummy interface. Under iptables and nftables the pod can additionally bind the ClusterIP. The IPVS-flavoured setup keeps working after the switch, and tidying it up means changing --cluster-dns on every kubelet plus a restart, which is a second and larger migration hiding inside the first. Give it its own change ticket, after the dataplane is stable.

Wrap-up

The objection you will actually have to argue is the scheduler one. KEP-5495 points out that many IPVS users have not realised the schedulers "aren't actually useful in Kubernetes": every node keeps its own IPVS table, so lc counts connections on that node alone, and conntrack pins every subsequent packet of a flow regardless. A team that says "we need least-connection balancing" has been getting a per-node approximation of random for years. That is a real cost of leaving IPVS only if you run one node.

Verify the switch instead of assuming it. After the canary rolls, ipvsadm -Ln returns nothing, ip link show kube-ipvs0 reports the device is gone, and nft list table ip kube-proxy prints a real ruleset. Hold the canary through one full traffic peak before touching the rest of the fleet, because the differences that hurt appear under load from a client nobody remembered.

The habit worth adopting today is tip 3: pin mode: explicitly in every cluster you own this week. It costs one line per cluster, and it decides whether you choose your dataplane or 1.40 chooses it for you.