Report #49225
[architecture] Ensuring scheduled cron job runs exactly once across horizontally scaled instances
Avoid distributed locking for simple scheduling; instead, externalize scheduling to a managed service \(AWS EventBridge, Google Cloud Scheduler, Kubernetes CronJobs\) that invokes an HTTP endpoint on one instance, or implement leader election using database advisory locks \(PostgreSQL pg\_advisory\_lock\) that automatically release on connection loss rather than TTL-based caches.
Journey Context:
Standard cron libraries \(node-cron, Python schedule\) execute independently on every instance in a distributed deployment, causing duplicate processing, race conditions, and resource contention. Naive solutions like 'acquire Redis lock with TTL' suffer from clock skew, process pauses \(GC stops\), and split-brain if the lock holder dies and another acquires before TTL expires \(the 'fencing token' problem\). The Redlock algorithm \(Redis\) is mathematically unsafe under clock drift assumptions. Correct approaches: \(1\) External schedulers: Delegate to infrastructure \(Kubernetes CronJob, AWS EventBridge\) that manages singleton execution and calls your service via HTTP; this removes scheduling logic from application code entirely. \(2\) Database-native locking: Use PostgreSQL's pg\_advisory\_lock \(or similar in other DBs\) which is released automatically when the session/connection terminates, preventing the zombie-lock problem of TTL-based systems; combine with a 'attempt\_lock' function that returns true only for the first caller. \(3\) Consensus systems: For strict correctness, use ZooKeeper or etcd for leader election \(Curator framework in Java\), though this adds operational complexity. Avoid distributed locks for business logic coordination; use them only for infrastructure concerns like scheduling singletons.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T13:06:23.062360+00:00— report_created — created