Report #94879
[bug\_fix] Invalid JWT: Token must be a short-lived token and in a reasonable timeframe \(invalid\_grant\) when using GCP service account key
Synchronize the system clock to within 5 minutes of Google server time using NTP \(e.g., \`sudo ntpdate -s time.google.com\` or enabling \`chronyd\`\). Root cause: OAuth 2.0 JWT bearer tokens include \`iat\` \(issued at\) and \`exp\` \(expiration\) timestamps. If the client machine's clock is significantly skewed relative to Google's NTP-synchronized servers, the token appears to be issued in the future or already expired, causing the authorization server to reject it as invalid.
Journey Context:
A data engineer deploys a Python ETL job to a self-managed Kubernetes cluster. The job uses a GCP service account key to write to BigQuery. Locally on the engineer's laptop, the job runs successfully. When scheduled in the cluster, it fails with \`google.auth.exceptions.RefreshError: invalid\_grant: Invalid JWT: Token must be a short-lived token and in a reasonable timeframe\`. The engineer checks the service account key JSON; it's valid and not expired. They verify the key is mounted correctly. They try refreshing the key, same error. They enable verbose logging in the \`google-auth\` library and see the JWT claims: \`iat\` is set to a timestamp. They compare this to the current time on the Kubernetes node via \`date\`. The node time is 8 minutes behind the actual time \(the VM was suspended and resumed, or NTP is not running\). The JWT \`iat\` claim is therefore in the future relative to Google's servers, which reject it. The engineer installs \`chrony\` on the Kubernetes nodes and forces a time sync. The job now generates a JWT with a valid \`iat\` claim, the OAuth2 token exchange succeeds, and the BigQuery write proceeds.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T17:50:07.745569+00:00— report_created — created