Cilium 1.19 Upgrade: Fix 12 Traps Before BGP Drops

Cilium 1.19 shipped on 24 February 2026, and 1.20.1 followed on 18 August, so most platform teams are doing the 1.18 to 1.19 hop right now, usually as a one-line Helm version bump. Three changes in that release rewrite behaviour you already depend on: cross-cluster policy selectors, the BGPv1 control plane, and a set of load-balancer flags that no longer exist. Two more are open regressions with no fix release. The same pre-flight discipline that saves you on a pnpm 12 upgrade applies here with much higher stakes, because the CNI is the component whose failure removes your ability to debug it.

Everything below comes from the upstream 1.19 upgrade guide, the Isovalent 1.19 release post, and five live GitHub issues linked at the claims they support. Run these in the maintenance window before helm upgrade, in roughly this order.

The 12 traps

1. Run the ClusterMesh policy inspector before you touch Helm. Cilium 1.19 flips policy-default-local-cluster to true, so a CiliumNetworkPolicy that selects pods by label now matches local-cluster endpoints only. The upgrade guide ships a command that lists exactly which of your policies change meaning:

    cilium clustermesh inspect-policy-default-local-cluster --all-namespaces

Empty output means the flip is a no-op for you. Any output is the list of cross-cluster paths that go dark the moment the new agent loads policy, which is a harsher version of the blast radius you plan for when you roll out default-deny egress.

1. Fix those policies with an explicit cluster selector, and put an expiry date on the escape hatch. Setting clustermesh.policyDefaultLocalCluster=false restores 1.18 behaviour, and the upgrade guide already marks that option for deprecation and removal. Buy one release with it if the inspector output is long, then convert each policy to an endpoint selector carrying the cluster label:

    - matchLabels:
        k8s:io.cilium.k8s.policy.cluster: remote-prod
        k8s:app: payments

File a ticket against the rollback flag with the 1.21 milestone in the title. Otherwise the next upgrade breaks the same traffic with nobody left who remembers why.

1. Move BGP to the v2 CRDs while you are still on 1.18, as its own change. 1.19 removed the CiliumBGPPeeringPolicy v1 API along with its agent implementation, so there is no in-place fallback once the chart lands. Migrate to CiliumBGPClusterConfig plus CiliumBGPPeerConfig plus CiliumBGPAdvertisement one node label at a time. If a v1 policy and a v2 cluster config both match a node, the legacy policy wins, an unsupported combination that produces peering you cannot reason about.
2. Check for ConflictingClusterConfigs after every label move. Two CiliumBGPClusterConfig objects matching the same node tear down all BGP sessions on that node, and the only signal is a status condition:

    kubectl get ciliumbgpclusterconfigs -o json \
      | jq -r '.items[] | select(.status.conditions[]?.type | test("Conflicting")) | .metadata.name'

Run it between node batches. A broad nodeSelector on the new config plus a leftover per-rack config is the usual way this fires.

1. Keep the BGP RBAC rule until the last 1.18 agent is gone. helm upgrade replaces the ClusterRole before the DaemonSet finishes rolling, and Cilium 1.19.3 dropped ciliumbgppeeringpolicies from it. Still-running 1.18.x agents then log:

    failed to list *v2alpha1.CiliumBGPPeeringPolicy: ciliumbgppeeringpolicies.cilium.io is forbidden:
    User "system:serviceaccount:kube-system:cilium" cannot list resource "ciliumbgppeeringpolicies" in API group "cilium.io"

Those agents fail to set up node BGP rather than degrading, so peers disappear mid-rollout (cilium#45802, open). Re-apply the rule immediately after the chart upgrade and delete it once every pod reports 1.19. This is one of the rare cases where widening a ServiceAccount's permissions temporarily is the correct call, so make the removal part of the same change ticket.

1. Read kube-proxy's mode before you roll a single node. On clusters where kube-proxy runs in IPVS mode, upgrading to 1.19.0 or 1.19.1 leaves agents at Init:0/6 with the config init container timing out against the API server:

    kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:'
    kubectl -n kube-system logs -l k8s-app=cilium -c config --tail=20   # dial tcp <apiserver>:443: i/o timeout

Clean installs of 1.19 on IPVS work; the upgrade path is the broken one (cilium#44464, open). Switch kube-proxy to iptables or nftables first, or plan a reboot per stuck node. Upgrade-only regressions like this are the same shape as the SELinux volume label conflict in Kubernetes 1.37: a fresh cluster never reproduces them, so a staging environment built from scratch will tell you nothing.

1. Grep the ConfigMap for the deleted service flags. --enable-node-port, --enable-host-port, --enable-external-ips, --enable-session-affinity and --enable-svc-source-range-check are gone in 1.19, folded into kube-proxy-replacement. So are --enable-custom-calls, --enable-ipv4-egress-gateway and --bpf-lb-proto-diff.

    kubectl -n kube-system get cm cilium-config -o json | jq -r '.data | keys[]' \
      | grep -E 'enable-(node-port|host-port|external-ips|session-affinity|svc-source-range-check|custom-calls|ipv4-egress-gateway)|bpf-lb-proto-diff'

Any hit is a Helm value or extraConfig entry someone set years ago. Delete it and set kubeProxyReplacement: true.

1. Leave unmanagedPodWatcher.intervalSeconds alone on the 1.19 line. The chart renders it as the string "15s" while the parser still wants an integer, and the pod dies at startup with option unmanaged-pod-watcher-interval: unable to cast "15s" of type string to int: strconv.ParseInt: parsing "15s": invalid syntax (cilium#44268). Drop the value from values.yaml for 1.19 and reinstate it after you reach 1.20.
2. Convert CiliumLoadBalancerIPPool to cilium.io/v2 as a pure re-apply, with the ranges frozen. The upgrade guide asks you to change apiVersion: cilium.io/v2alpha1 to cilium.io/v2 before upgrading. Make that edit and nothing else, because pool updates have been observed reassigning addresses across every service bound to the pool (cilium#40358), which rewrites the IPs your DNS records and firewall rules point at. Snapshot first, diff after:

    kubectl get svc -A -o json \
      | jq -r '.items[] | select(.status.loadBalancer.ingress) | "\(.metadata.namespace)/\(.metadata.name) \(.status.loadBalancer.ingress[0].ip)"' > lb-before.txt

1. Count advertised routes after the BGP cutover, not established sessions. Under the v2 API, externalTrafficPolicy: Local LoadBalancer IPs are advertised from every selected node regardless of local endpoints, reported against 1.19.1 through 1.20.0 and working on 1.18.7 (cilium#44803, open). Sessions come up, routes look healthy, and the upstream router picks a control-plane node with no backing pod:

    cilium bgp routes advertised ipv4 unicast | grep 203.0.113.10

More advertising nodes than endpoints means you are hitting it. Pin those services to externalTrafficPolicy: Cluster until the fix lands and accept the loss of source IP preservation, which costs you client-IP-based rate limiting and audit fidelity on those paths.

1. Sweep the policy CRDs for Requires fields and . DNS patterns in one pass. 1.19 enforces that FromRequires and ToRequires are empty, so a policy carrying them is rejected on apply. Separately, in a toFQDNs matchPattern used to behave like *, and it now matches multiple subdomain levels, which silently widens egress you already approved.

    kubectl get cnp,ccnp -A -o json \
      | jq -r '.items[] | select(tostring | test("[Rr]equires|\\*\\*\\.")) | "\(.kind) \(.metadata.namespace)/\(.metadata.name)"'

1. Re-enable mutual auth explicitly, or it stops guarding anything. mesh-auth-enabled (Helm authentication.enabled) is disabled by default in 1.19. If any policy carries authentication: mode: required, confirm the flag survived your values merge, because the datapath drops matching packets until a handshake completes and a half-configured state surfaces as unexplained drops with no warning:

    kubectl get cnp,ccnp -A -o json | jq -r '.items[] | select(tostring | test("\"mode\":\"required\"")) | .metadata.name'

Then hold at 1.19 for a release. 1.20 removes proxylib and Kafka-aware L7 policies, the v2alpha1 CiliumNodeConfig, and --k8s-api-server, and it requires Gateway API v1.6.1. Two CNI-breaking hops in one window leaves you unable to bisect.

FAQ

Do I have to migrate CiliumBGPPeeringPolicy before upgrading to 1.19? Yes. The v1 API and its agent implementation are removed in 1.19, so the migration has to happen on 1.18 while both code paths exist.

What does policy-default-local-cluster actually change? It scopes label-based endpoint selectors in CiliumNetworkPolicy to the local cluster by default. Policies that previously matched pods in a meshed cluster stop matching them until you add an explicit k8s:io.cilium.k8s.policy.cluster label to the selector.

Can I go straight from Cilium 1.18 to 1.20 and skip 1.19? Upstream supports one minor version at a time, and skipping stacks the BGPv1 removal, the ClusterMesh policy flip, and the 1.20 proxylib and CiliumNodeConfig removals into a single change with no way to attribute a failure.

Why are my Cilium pods stuck at Init:0/6 after the upgrade? On clusters running kube-proxy in IPVS mode, the config init container times out reaching the API server on 1.19.0 and 1.19.1 (cilium#44464). Check the kube-proxy mode first; a node reboot clears individual stuck nodes.

Does kubeProxyReplacement replace --enable-node-port? It does. The five per-feature service flags are deleted in 1.19 and their behaviour is controlled by kube-proxy-replacement, so a stale --enable-node-port in cilium-config is now an unknown option.

Wrap-up

The habit worth keeping: every Cilium minor upgrade gets a dry run against the policy and BGP CRDs before the chart version changes. Ten minutes of jq over cilium-config, cnp, ccnp and the BGP objects tells you which of these twelve apply to your cluster. The two open regressions, IPVS init timeouts and externalTrafficPolicy: Local over-advertisement, decide whether you go to 1.19 this month or wait, and neither of them appears in the release notes.