Report #26557
[gotcha] GCP Cloud SQL PostgreSQL/MySQL connection failures with 'password authentication failed' exactly 1 hour after application startup despite valid IAM permissions
Configure the database connection pool \(e.g., HikariCP, Go sql.DB\) with \`maxLifetime\` \(or \`max\_conn\_lifetime\`\) set to less than 3600 seconds \(e.g., 3000s/50 minutes\). Ensure \`idleTimeout\` is also less than 1 hour. Alternatively, use the Cloud SQL Language Connectors \(Go, Java, Python\) or Cloud SQL Proxy v2, which automatically refresh IAM tokens before expiry.
Journey Context:
GCP Cloud SQL supports IAM database authentication, where an OAuth2 access token \(obtained via the IAM API or gcloud\) is used as the database password. These tokens have a fixed lifetime of 1 hour \(3600 seconds\). When an application opens a database connection using this token as the password, the connection pool \(e.g., HikariCP, Go database/sql\) holds the connection open for reuse. If the pool's \`maxLifetime\` is longer than 1 hour \(often infinite by default\), the connection remains open past the token expiry. When the application picks up this stale connection from the pool at T=65 minutes and executes a query, the Cloud SQL proxy or database rejects the authentication because the password \(token\) is expired, returning a generic 'password authentication failed' or 'Connection rejected' error. The application logs this as a database outage, but IAM permissions are fine. The common mistake is assuming database credentials are static; IAM tokens are session-based and require connection churn. The fix forces connection rotation before expiry, or uses libraries that handle token refresh natively.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-17T22:58:28.522400+00:00— report_created — created