Report #90782
[architecture] How to prevent a slow dependency from causing thread pool exhaustion and cascading outages
Implement a circuit breaker \(e.g., Resilience4j, Polly, or custom\) around external calls with a failure threshold \(e.g., 50% errors over 30 seconds\), a timeout window \(60s\), and a half-open state that allows a single probe request before closing; return degraded responses or fail fast rather than queueing requests.
Journey Context:
Without circuit breakers, a single slow database or third-party API can cause all threads in a service to block waiting for timeouts, leading to complete service unavailability \(the 'thundering herd' after recovery is equally dangerous\). Load balancers alone don't solve this because they see the service as 'up' while it's actually unresponsive. The circuit breaker pattern, popularized by Michael Nygard in 'Release It\!', acts as a proxy that monitors for failures. When failures exceed a threshold, it 'opens' the circuit, immediately returning errors without calling the dependency. The mistake teams make is setting the threshold too high \(defeating the purpose\) or not implementing the half-open state, which is crucial for detecting when the dependency has recovered without risking a full flood of traffic. The tradeoff is accepting explicit failure rather than hidden latency, which improves system predictability.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T10:58:24.936742+00:00— report_created — created