Kubernetes service account tokens: projection, rotation, and lifecycle

What this article is for

Read this if you upgraded past Kubernetes 1.24 and discovered that pods, CI/CD jobs, or external integrations that used to read a token from a Secret now have nothing to read. The mechanism that produced those tokens has been gone for several releases, and the replacement behaves differently in ways that matter for how you wire up authentication.

This is a concept article. It explains what projected tokens are, why the model changed, and how the moving parts fit together. For the procedural counterparts (creating a token for an external tool, configuring automountServiceAccountToken, running a pipeline against the API), the article links to the relevant how-to and reference docs at the end.

What service account tokens are and why pods need them

A service account is the identity a pod uses to authenticate to the Kubernetes API server. The pod does not log in with a username and password. It presents a signed JSON Web Token that the API server validates against keys it already trusts.

Every pod runs as exactly one service account. If the pod spec does not name one, it runs as default in its namespace. The default service account has no useful permissions out of the box, but the token mechanism still applies: a token is generated, mounted into the pod, and rotated on schedule.

Tokens are also useful outside pods. CI/CD pipelines, dashboards, monitoring agents, and ad-hoc kubectl calls authenticate as service accounts when they cannot use a user identity. Whatever the caller, the API server's check is the same: is the token valid, and what subject does it represent? RBAC then decides what that subject is allowed to do, which is the topic of the RBAC concept article.

Before Kubernetes 1.24: Secret-based tokens (and why they were dropped)

In Kubernetes 1.23 and earlier, every service account got an automatically generated Secret of type kubernetes.io/service-account-token. The tokens controller created it on the fly when you ran kubectl create serviceaccount. The Secret contained a token that:

• Did not expire
• Was not bound to any pod, node, or audience
• Stayed valid as long as the Secret existed
• Was usable from anywhere, by anyone who obtained it

That model worked, but it failed badly under stress. A token leaked from a CI log was valid forever. A token copied off a developer's laptop kept working after they left. Tokens propagated through container registries, through screen recordings, through paste history. There was no rotation, no expiry, and no way to scope the blast radius.

KEP-2799 addressed this. As of Kubernetes 1.24, the API server no longer auto-creates Secret-based tokens for new service accounts. The feature gate LegacyServiceAccountTokenNoAutoGeneration is on by default, and the behavior went GA in 1.26. You can still create a Secret of type kubernetes.io/service-account-token by hand if a tool genuinely needs a long-lived token, but it is no longer the default and no longer expected.

Since Kubernetes 1.22: projected service account tokens

The replacement is the bound service account token, delivered via a projected volume. It went GA in Kubernetes 1.22 under the feature gate BoundServiceAccountTokenVolume, which means by the time the legacy mechanism was removed in 1.24, the replacement had already been default for two releases.

A projected token is materially different from a Secret-based token in five ways:

1. Time-bound. Tokens have an exp claim. Default expiry is one hour, and the TokenRequest API refuses to issue tokens with a duration shorter than 10 minutes.
2. Audience-bound. Each token names the audiences it is valid for in the aud claim. A token issued for one audience is rejected by API servers expecting another, which is what makes it safe to issue cluster-bound tokens for external tools without those tokens being usable elsewhere.
3. Object-bound. A pod-mounted token is bound to that specific Pod object's UID. If the Pod is deleted, the token becomes invalid 60 seconds after the deletion timestamp, even if the JWT is otherwise still within its validity window.
4. Automatically rotated. The kubelet refreshes the token before it expires. Specifically, the kubelet starts trying to rotate the token once the token is older than 80 percent of its TTL or older than 24 hours, whichever comes first.
5. Mounted at the same path. For backwards compatibility, the projected token is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. Code written against the old Secret-based location keeps reading the file successfully. The file's contents change, the file's path does not.

The cluster operator sets the maximum allowed token lifetime via the --service-account-max-token-expiration flag on the API server. The default is one hour. A pod or kubectl create token caller can request a shorter duration but cannot exceed the cluster maximum.

What a projected volume looks like

The kubelet writes the projected volume into every pod that has automountServiceAccountToken enabled (which is most pods, by default). You rarely need to declare it manually, but the equivalent volume looks like this:

volumes:
  - name: kube-api-access
    projected:
      sources:
        - serviceAccountToken:
            # Bound to the Kubernetes API server audience by default
            audience: ""
            # Token TTL in seconds; max is the API server's flag value
            expirationSeconds: 3600
            path: token
        - configMap:
            name: kube-root-ca.crt
            items:
              - key: ca.crt
                path: ca.crt
        - downwardAPI:
            items:
              - path: namespace
                fieldRef:
                  fieldPath: metadata.namespace

The kubelet attaches this volume automatically. The token rotates in place: applications that re-read the file on each request always see a fresh token without the pod restarting.

The 80 percent rotation rule in practice

Because the kubelet refreshes the token at 80 percent of its TTL, an application that caches the token in memory will eventually use a token that has been replaced on disk. Most clients handle this by re-reading /var/run/secrets/kubernetes.io/serviceaccount/token on every request. The official Kubernetes client libraries (client-go, the Python client, the Java client) do this automatically. If you write your own client, do not cache the token across requests.

Disable token automounting on pods that do not need API access

A pod that never calls the Kubernetes API still gets a token mounted by default. That mount is an attack surface: any process inside the pod (including a compromised dependency) can read the token from disk and authenticate to the API as the service account.

The fix is to disable the automount. Two places to set it, with the pod spec winning when both are set:

# Option 1: at the ServiceAccount, applies to every pod that uses it
apiVersion: v1
kind: ServiceAccount
metadata:
  name: frontend
  namespace: production
automountServiceAccountToken: false

# Option 2: at the pod spec, overrides the ServiceAccount setting
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  serviceAccountName: frontend
  automountServiceAccountToken: false

If a pod genuinely needs API access for one specific operation, prefer mounting the token only for that pod and leave the ServiceAccount default off. The Configure Service Accounts for Pods documentation confirms that the pod spec takes precedence when both fields are set.

Generate tokens manually for CI/CD and external tooling

External tools that need to authenticate as a service account no longer get a token from a Secret. Two supported patterns:

Pattern 1: short-lived tokens via TokenRequest

# Issue a token valid for one hour for the ci-deployer ServiceAccount
# in the production namespace. The client only sees this token; it is
# not stored anywhere on the cluster.
kubectl create token ci-deployer \
  --namespace=production \
  --duration=1h

This is the recommended pattern for CI/CD, scripts, and any tool that runs to completion. The duration cap is the API server's --service-account-max-token-expiration flag (default one hour). For a worked example of wiring this into a pipeline, see Kubernetes least-privilege RBAC patterns for CI/CD.

Pattern 2: long-lived token via a Secret

For tools that genuinely cannot deal with short-lived tokens, manually create a Secret of type kubernetes.io/service-account-token:

apiVersion: v1
kind: Secret
metadata:
  name: ci-deployer-token
  namespace: production
  annotations:
    kubernetes.io/service-account.name: ci-deployer
type: kubernetes.io/service-account-token

The control plane populates the data field with a token after the Secret is created. This token does not expire, so treat it the same way you would a permanent API key: store it in a secrets manager, rotate it on a schedule, and revoke it (delete the Secret) if it is suspected of leaking.

This pattern is supported, but it is the exception. Most tools support short-lived tokens, and most that do not should be on the migration list.

Migrate from Secret-based tokens to projected tokens

The migration path looks the same for nearly every cluster:

1. Find legacy auto-generated tokens. The cluster controller adds the kubernetes.io/legacy-token-last-used label to Secret-based tokens it has seen used. Listing Secrets with that label tells you which legacy tokens are still in active use:

   # List every legacy service account token Secret cluster-wide,
   # showing the last-used date so you know which ones are still active.
   kubectl get secrets --all-namespaces \
     --field-selector type=kubernetes.io/service-account-token \
     -L kubernetes.io/legacy-token-last-used

2. Identify the consumer. A token in a Secret was being read by something. Trace it back: a CI pipeline, a Helm chart, a custom controller. The exact path it reads is usually in the consumer's documentation or environment variables.

3. Replace with kubectl create token or a projected volume. Pipelines switch to issuing fresh tokens per run. In-cluster controllers switch to reading the projected token from /var/run/secrets/kubernetes.io/serviceaccount/token, which is what they were doing in 1.22 anyway.

4. Set automountServiceAccountToken: false on every ServiceAccount whose pods do not call the API.

5. Delete the Secret-based token once nothing reads it for a few days. The cluster will eventually do this automatically (next section), but you should not wait if you can verify the consumer is gone.

The --service-account-extend-token-expiration API server flag, default true, is the migration safety net. When this flag is on, tokens issued through projected volumes that are about to expire get a short extension if the workload appears to be using legacy behavior, so existing pods that have not been updated to handle rotation continue to authenticate during the upgrade window. The flag exists specifically for migrations and can be disabled once you are confident no legacy clients remain.

Kubernetes 1.30: automatic cleanup of unused legacy tokens

Even after the auto-creation of Secret-based tokens stopped, plenty of clusters still had legacy Secrets sitting around from the pre-1.24 days. Some were in active use; many were not. The control plane cannot tell the difference between a token nobody reads any more and a token that some weekly batch job will need next Tuesday.

The legacy service account token cleaner addresses this. It went GA in Kubernetes 1.30 under the feature gate LegacyServiceAccountTokenCleanUp. The mechanics:

1. The control plane records the date of last use on each legacy token Secret via the kubernetes.io/legacy-token-last-used label.
2. After a Secret has gone unused for the configured period (default one year, set via --legacy-service-account-token-clean-up-period on the kube-controller-manager), the cleaner adds the kubernetes.io/legacy-token-invalid-since label with the current date and the token stops being accepted.
3. After another full period (another year by default) since invalidation, the cleaner deletes the Secret entirely.

Two safety properties matter for operators:

• Tokens currently mounted by pods are never marked invalid. The cleaner skips Secrets that any pod still references. A long-running workload that has the Secret mounted will not have its token invalidated under it.
• Invalidation is reversible. If a tool was supposed to use the token and fails authentication after invalidation, an admin can remove the kubernetes.io/legacy-token-invalid-since label to put the Secret back in service while migration completes. This is a temporary measure, not a long-term fix.

After 1.30, you can stop manually inventorying legacy tokens. The cluster does it for you. But that is exactly why you should make sure your monitoring catches authentication failures: if a tool has been silently relying on a legacy token, the failure mode is a 401 from the API server, not a warning ahead of time.

Debug token-related authentication failures

When a workload that previously worked starts returning 401 Unauthorized, the most common causes are predictable:

• Cluster upgraded past 1.24 and the workload was reading a Secret-based token that no longer exists. Inspect the Secret: if the data.token field is missing or empty, the auto-generation mechanism did not create one. Generate a fresh token explicitly or migrate the workload to read the projected token.
• Token mounted by the kubelet has rotated and the application cached the old one. Check the application's token-handling code. Re-read the file on every request; do not cache.
• Token was issued for the wrong audience. A token issued via kubectl create token --audience=https://my-other-system will be rejected by the API server because the audience does not match. Use the default empty audience for API server authentication, and a non-default audience only when an external system explicitly requires it.
• Pod was deleted and the token became invalid. The 60-second grace period after deletion is short by design. Workloads that need to do graceful shutdown work that includes API calls should issue those calls before the deletion timestamp, not after.
• Legacy token marked invalid by the 1.30 cleaner. Check the Secret for the kubernetes.io/legacy-token-invalid-since label. If present, the cleaner has invalidated it. Migrate the consumer or, as a temporary measure, remove the label.

For deeper RBAC-related authorization failures (403 Forbidden), the RBAC debugging walkthrough covers how to use kubectl auth can-i to identify which permission is missing.

What service account tokens are NOT

The model is easy to misunderstand. Three claims are wrong, and worth being precise about.

"Service account tokens never expire." This was true for legacy Secret-based tokens. It is not true for projected tokens, which expire by default after one hour and rotate automatically. If your code assumes a service account token is a long-lived credential, it is reading a different system than the current one.

"Every pod needs a service account token." A pod that does not call the Kubernetes API does not need a mounted token. Setting automountServiceAccountToken: false removes a real attack path with no behavioral cost. The default is on, but the default is conservative, not correct.

"You can reuse a token across clusters." Tokens are signed by the cluster's private key and validated by that cluster's public key. A token from cluster A is a piece of unrelated bytes to cluster B. Beyond that, audience binding makes even within-cluster reuse limited: a token issued for one audience cannot authenticate against another. There is no portable cluster-spanning service account token, by design.

"automountServiceAccountToken: false breaks workload identity." It does not. Cloud workload identity systems on EKS, GKE, and AKS use a separate projected volume for their cloud-bound token. Disabling the standard mount only removes the Kubernetes API token, which is what you want for a pod that authenticates to a cloud provider but never to the Kubernetes API directly.

"A Secret of type kubernetes.io/service-account-token is the same as the projected token." They are different mechanisms with different security properties. The Secret-based token is long-lived, has no audience binding, and is not bound to any pod's lifecycle. The projected token is short-lived, audience-bound, pod-bound, and rotated. They both authenticate to the same API server, but only the projected token has the safety properties that came in with KEP-1205.

When to escalate

If a service-account token issue does not resolve through the steps above, collect the following before asking for help:

• The exact 401 or 403 error message from the API server (full text, including any WWW-Authenticate header)
• Output of kubectl describe sa <name> -n <namespace> and kubectl get secrets -n <namespace> -l kubernetes.io/service-account-name=<name>
• For pod-side problems: kubectl describe pod <name> and the contents of /var/run/secrets/kubernetes.io/serviceaccount/token (decoded with jwt.io to inspect the claims, never to debug a real production token in a public site)
• Kubernetes version (kubectl version)
• API server flags relevant to tokens: --service-account-issuer, --service-account-max-token-expiration, --service-account-extend-token-expiration, --service-account-key-file
• Whether the cluster has any custom authentication or authorization webhooks that might reject the token after the API server validates it

This information lets someone separate "the token was rejected by the API server" from "the token was accepted but RBAC denied the action" from "an admission webhook ate the request before it reached the authorization layer". They are different problems with different fixes.

Where to go next

• Kubernetes RBAC covers what an authenticated service account is allowed to do.
• Kubernetes secrets management compares Sealed Secrets, External Secrets Operator, and Vault for production secret handling.
• Workload identity for Kubernetes explains how the projected token model extends to cloud provider authentication on EKS, GKE, and AKS.