Preventing Cascading Failures with Connection Read and Deadline Timeouts in Distributed Systems
Video tutorial: Preventing Cascading Failures with Connection Read and Deadline Timeouts in Distributed Systems
Every remote call needs a deadline, because a call that never returns holds a worker, a connection and a user's patience indefinitely, and one slow dependency with no deadline will consume the whole pool and take down callers that do not even use it. Having accepted that, the value is a design decision with evidence behind it, and the evidence is the measured distribution of how long that call actually takes. Set the deadline above the slow tail of successful calls and it only fires on genuine trouble. Set it below the tail and you manufacture failures: requests that would have succeeded are abandoned, the work is often done anyway on the other side, and if the caller retries you have doubled the load on a dependency that was already struggling. So the rule is to derive the number from a high percentile of observed successful calls with a margin, review it whenever the dependency changes, and record what it was derived from, because an inherited timeout is a number nobody can defend. Two consequences follow. A deadline that fires leaves the outcome unknown, so the operation behind it must be safe to repeat. And the value you derive here is spent against an end-to-end budget that a separate Concept owns. You can now propose a timeout value with the measurement it came from.
Video tutorial: Preventing Cascading Failures with Connection Read and Deadline Timeouts in Distributed Systems