Bound ServiceAccount Tokens: 9 Tips to Kill Static Ones

Q: Why does my Pod get a 401 right after migrating to projected tokens?

Usually an audience mismatch rather than an RBAC gap. Bound tokens carry an aud claim and the API server rejects any token whose audience is not listed in its --api-audiences, so a token minted for https://vault.internal is refused by the API server because it was never addressed to it. Decode the payload and read the aud claim before touching roles.

Q: Does deleting a Pod invalidate a token bound to it?

Yes, immediately. A token created with --bound-object-kind=Pod stops being accepted the instant that Pod is deleted, even when its expiry has not been reached. That is why a token leaked from a job that has since terminated is already dead, and it is the containment property a static Secret never had.

Q: How do I reactivate a token already labeled legacy-token-invalid-since?

Delete that one label from the Secret and the token is accepted again, which buys you time to migrate the integration properly. Treat it as a stopgap only: the LegacyServiceAccountTokenCleanUp controller will relabel it, so the real fix is cutting the consumer over to the TokenRequest API.

Q: Why does Istio report does not match claim after I recreated a ServiceAccount?

Bound tokens pin the UID of the ServiceAccount they were issued against, not just its name. Deleting and recreating a ServiceAccount with the same name produces a new UID, which invalidates every token minted against the old one, and mesh sidecars surface that as a claim mismatch instead of a plain 401.

Move Kubernetes off non-expiring ServiceAccount secrets to bound, projected tokens via the TokenRequest API, and fix the 401s the migration triggers.

The non-expiring kubernetes.io/service-account-token Secret is the most over-scoped credential in most clusters. It never expires, it authenticates from anywhere on the internet, and nothing rotates it. Kubernetes has spent five releases retiring it: v1.24 stopped auto-minting it, and v1.30 promoted the cleaner that invalidates the leftovers to stable. The replacement is a time-bound, audience-scoped JWT from the TokenRequest API, tied to a Pod's lifetime, and it behaves differently enough that an integration working last quarter can start throwing 401s after an upgrade.

These tips are for platform and security engineers who govern non-human identity on Kubernetes and want to retire static tokens without an outage. This is the concrete mechanism the SPIFFE-and-OAuth identity essay argued for, and the follow-through to non-human identity governance in the field. The same instinct drove dropping static keys in GitHub Actions with OIDC: every credential you cannot delete is one you have to defend.

TL;DR: Inventory every remaining kubernetes.io/service-account-token Secret, triage it by its legacy-token-last-used label, and cut live consumers over to the TokenRequest API before the v1.30 cleaner invalidates them for you. Bound tokens expire, carry an audience, and die with the Pod they were issued for. The 401s you hit during the migration are almost always an audience mismatch or a node-audience authorization gap, not broken RBAC.

How does a bound ServiceAccount token actually get issued?

A projected token is not a stored object you can read back. The kubelet asks the control plane for one on the Pod's behalf, writes it into the mount, and refreshes it on a clock. The audience you asked for is the same value the receiving server checks, which is where most migrations break.

Which legacy tokens do I still have, and which are alive?

Tip 1: inventory every legacy token Secret before the cleaner finds them for you. The LegacyServiceAccountTokenCleanUp controller went GA in v1.30. It labels any auto-generated token Secret unused for a year with kubernetes.io/legacy-token-invalid-since, and the token stops being accepted the moment that label lands. Find these on your schedule, not as a surprise 401 in production. List every remaining secret-based token cluster-wide:

  kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

Tip 2: read the last-used label to separate live tokens from fossils. The control plane stamps each legacy token with kubernetes.io/legacy-token-last-used, a date at day granularity. That label is your triage: a Secret used this week is a live integration you have to cut over, and one untouched for months is a fossil you can delete now.

  kubectl get secret <name> -o jsonpath='{.metadata.labels.kubernetes\.io/legacy-token-last-used}'

If a Secret already carries legacy-token-invalid-since and you need it working again immediately, delete that one label to reactivate it, then migrate it properly. The cleaner will relabel it otherwise.

How do I mint a short-lived token the right way?

Tip 3: issue tokens on demand with kubectl create token, never by reading a Secret. For any human or script that needs to act as a ServiceAccount, mint a short-lived token through the TokenRequest API rather than pulling a static one with kubectl get secret ... -o jsonpath. It respects RBAC, carries a real expiry, and leaves no long-lived artifact on disk or in etcd.

  kubectl create token deploy-bot -n ci --duration=30m

Tip 4: know the 10-minute floor and the 1-hour default. The TokenRequest API refuses to issue a token shorter than 10 minutes: ask for --duration=5m and the server clamps or rejects it. Specify nothing and the default lifetime is one hour. Do not design a 90-second credential, because the platform will not give you one. For genuinely ephemeral use, 10 minutes is the floor you build against.
Tip 5: bind the token to the object that should kill it. --bound-object-kind accepts Pod, Secret, and Node. A token bound to a Pod becomes invalid the instant that Pod is deleted, even if its clock has not run out, so a token leaked from a since-terminated job is already dead. That is the property static Secrets never had.

  kubectl create token app -n prod \
    --bound-object-kind=Pod --bound-object-name=app-7d9f --duration=15m

What should a Pod mount instead of a Secret?

Tip 6: in Pods, mount a projected token and set expirationSeconds deliberately. Pods get their token through a projected volume, not a Secret mount. The kubelet rotates it automatically once the token passes 80% of its TTL or 24 hours, whichever comes first. Set expirationSeconds to match how long a single request path actually needs, not "forever":

  volumes:
  - name: token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          audience: vault
          expirationSeconds: 3600

Why does my workload get a 401 after the cutover?

Tip 7: the most common post-migration 401 is an audience mismatch, not a permissions problem. Bound tokens carry an aud claim, and the API server rejects any token whose audience is not in its --api-audiences. A token minted with --audience=https://vault.internal is refused by the API server itself, because it was never addressed to it. Before you touch RBAC, decode the token and check where it is actually pointed:

  kubectl create token app -n prod | cut -d. -f2 | base64 -d 2>/dev/null | jq .aud

Tip 8: understand the node-audience gate before wiring cloud or Vault audiences. By default a kubelet may only request tokens for audiences already referenced by Pods on that node. If your CSI driver or sidecar asks for a fresh audience the node has never mounted, the request fails until you grant it through an RBAC rule carrying the request-serviceaccounts-token-audience verb. Teams adding a new external audience mid-cluster-life hit this wall and misread it as an app bug. It is an authorization gap on the node, not your code.

What still breaks after the migration looks finished?

Tip 9: make consumers re-read the token file, and do not trust a recreated ServiceAccount's old token. Two production traps hide in the details here. First, because the kubelet rotates the projected file in place, a client that reads the token once at startup authenticates fine for an hour and then starts failing, so reload from the mount path on every request.

Second, bound tokens pin the ServiceAccount's UID, so deleting and recreating a ServiceAccount with the same name invalidates every token already issued against the old UID. Istio and other mesh sidecars surface exactly this as a does not match claim error. Watch kubernetes/kubernetes#138689 too: a transient TokenRequest or NodeAuthorizer failure during a control-plane restart can leave a projected token file stale, so make token-refresh failures an alert rather than a silent log line.

Wrap-up

Run the --field-selector type=kubernetes.io/service-account-token inventory today, sort by the legacy-token-last-used label, and cut the live ones over to TokenRequest before the v1.30 cleaner turns "unused for a year" into an unscheduled 401. Bound tokens are the version of a credential you do not have to defend, since expiry, audience, and Pod lifetime do the containment for you.

Move identity on the same clock as the rest of your controls, including a default-deny egress policy, so a leaked token cannot phone home even in the minutes before it dies. If you would rather have the inventory and the cutover run against your own clusters by someone who has done it, that is work I take on.

FAQ

What is the shortest lifetime a bound ServiceAccount token can have? Ten minutes. The TokenRequest API refuses anything below that floor, so a request for 5m is clamped or rejected outright, and asking for nothing gives you the one hour default. Design against the floor rather than a credential measured in seconds.

Why does my Pod get a 401 right after migrating to projected tokens? Usually an audience mismatch rather than an RBAC gap. The API server rejects any token whose aud claim is not listed in its --api-audiences, so decode the payload and read the audience before you start editing roles.

Does deleting a Pod invalidate a token bound to it? Yes, immediately. A token created with --bound-object-kind=Pod stops being accepted the instant that Pod is deleted, even when its expiry has not been reached, which is why a token leaked from a terminated job is already dead.

How do I reactivate a token already labeled legacy-token-invalid-since? Delete that one label and the token is accepted again. Treat it as a stopgap: the cleaner will relabel it, so the real fix is moving the consumer to the TokenRequest API.

Why does Istio report does not match claim after I recreated a ServiceAccount? Bound tokens pin the ServiceAccount's UID, not just its name. Recreating it with the same name produces a new UID and invalidates every token issued against the old one.