In a single-process application, success and failure can seem relatively clear. A function returns a result or raises an error. A transaction commits or rolls back.
Distributed systems are less considerate.
A request may reach a service even when the response never returns. A database write may succeed just before the connection drops. One dependency may be healthy from one availability zone and unreachable from another. A queue may accept work faster than consumers can process it.
The system is neither fully working nor fully broken. It is partially failing.
Resilient architecture begins by treating this condition as normal.
A timeout does not reveal what happened
Imagine Service A asks Service B to create a record. Service B creates it, but the response is lost. Service A hits its timeout.
From Service A’s perspective, the outcome is unknown—not necessarily a failure.
Retrying immediately may create a duplicate record. Refusing to retry may leave the caller believing an operation failed when it actually succeeded. Extending the timeout may simply tie up more resources while the dependency remains unhealthy.
This ambiguity is fundamental to network communication. It cannot be eliminated with a framework.
The operation needs a design that can tolerate uncertainty. Common approaches include an idempotency key, a client-generated operation identifier, a status endpoint, or a durable workflow that can resume after interruption.
The important shift is conceptual: a timeout is not a business result. It is evidence that the caller stopped waiting.
Put a deadline on every remote call
A remote call without a timeout allows an unhealthy dependency to consume threads, sockets, memory, and connection-pool entries indefinitely.
Every network boundary should have a deadline based on the work being performed and the caller’s own time budget.
Suppose an API must respond within two seconds but calls three services sequentially. Assigning a two-second timeout to every dependency cannot satisfy the overall objective. The time budget must account for the entire call chain, including processing, network latency, queuing, and any permitted retries.
Timeouts should also be observable. Teams need to know which dependency timed out, how long the caller waited, and whether the operation later completed.
A timeout protects the caller. It does not repair the dependency.
Retry selectively and within a budget
Retries are useful for transient conditions such as a brief connection interruption or a temporarily unavailable instance. They are harmful when the error is permanent or the dependency is already overloaded.
A sound retry policy normally includes:
- A small maximum number of attempts
- Exponential or progressive backoff
- Jitter to prevent synchronized retries
- A total time budget
- Explicitly retryable error categories
- Idempotent operation semantics
Retries should not be stacked blindly at every layer.
If an HTTP client retries three times, an application service retries three times, and a job runner retries three times, one failed operation can produce 27 downstream attempts. During an incident, that amplification consumes the capacity needed for recovery.
Ownership of retries should be deliberate. Often, the layer with the clearest understanding of the operation’s semantics is best positioned to decide.
Make repeated commands safe
A resilient system assumes that messages and requests may arrive more than once.
Idempotency means that repeating an operation produces the same intended outcome rather than duplicating its effect. Reading a record is naturally idempotent. Charging a card, sending a notification, or creating an order may not be.
One practical design assigns each command a stable identifier. Before performing the side effect, the receiving service checks whether that identifier has already been processed. The result can be stored and returned for repeated requests.
The implementation details matter. Checking for an identifier and then writing the result in separate, unprotected steps can still create races. The deduplication record and business change may need to share a transaction or a uniqueness constraint.
Idempotency does not mean the code runs only once. It means multiple executions do not corrupt the business outcome.
Stop failure from spreading
A service can be healthy internally and still become unavailable by exhausting resources while waiting on another system.
Bulkheads reduce this risk by separating capacity. Calls to one dependency might use a dedicated connection pool, queue, worker group, or concurrency limit. If that dependency slows down, it consumes only its allocated portion rather than every available resource.
Circuit breakers provide another form of containment. After failures cross an explicit threshold, the circuit stops sending normal traffic for a period. A limited probe can later determine whether the dependency has recovered.
The circuit breaker is not merely a code pattern. Its thresholds and fallback behavior need to reflect the business operation.
Opening a circuit for an optional recommendation service might remove recommendations temporarily. Opening one for an authorization service may require rejecting the operation entirely. The correct degraded behavior depends on the risk of proceeding without the dependency.
Use backpressure before capacity disappears
Queues help absorb bursts, but they do not create unlimited capacity.
When producers consistently create work faster than consumers can process it, the queue delays failure rather than preventing it. Latency grows, stored messages become stale, and recovery takes longer even after incoming traffic falls.
Backpressure makes limited capacity visible to upstream components. Depending on the system, that might mean:
- Rejecting work with a retryable response
- Slowing producers
- Applying per-client quotas
- Dropping low-value telemetry
- Coalescing repeated updates
- Prioritizing critical commands
- Scaling consumers within a defined limit
Throttling only at the public gateway may not be enough. Internal fan-out, retries, scheduled jobs, and shared dependencies can also create overload. Current Azure guidance recommends considering throttling across the call chain rather than only at ingress. Microsoft Azure Well-Architected Framework
Degrade by business value
Graceful degradation is often described too vaguely. A system does not become resilient simply because an exception is caught and an empty response is returned.
Useful degradation starts by classifying capabilities.
For an online shop, checkout may be critical while personalized recommendations are optional. For a vehicle system, a diagnostic upload might be deferred while local control behavior must remain deterministic. For a trading platform, delayed reference data may be tolerable in one workflow and dangerous in another.
Possible degraded behaviors include serving cached data, returning a reduced response, postponing noncritical work, disabling an expensive feature, or switching to a carefully defined safe mode.
The degraded state must be observable. Otherwise, teams may unknowingly operate a partially functional system for days.
Design recovery, not only protection
Resilience mechanisms reduce damage, but recovery still needs a plan.
After a dependency returns, queued work may rush toward it and trigger another failure. Circuits may close simultaneously. Cached entries may all expire together. Operators may need to reconcile operations that ended in an unknown state.
Recovery should therefore be controlled:
- Probe the dependency with limited traffic.
- Restore capacity gradually.
- Keep retries bounded.
- Monitor latency and saturation, not only error rate.
- Reconcile incomplete business operations.
- Confirm that degraded features have returned.
- Preserve evidence for incident analysis.
A system is not recovered merely because every process is running.
Test failures as behaviors
A resilience design that has never been exercised is an assumption.
Testing should cover more than terminating a container. Real incidents include slow responses, malformed payloads, expired credentials, connection-pool exhaustion, duplicated messages, clock differences, partial packet loss, unavailable name resolution, and schema incompatibility.
Start with focused experiments in controlled environments:
- Delay one dependency beyond its timeout.
- Return errors for a percentage of requests.
- Deliver the same command twice.
- Stop a consumer while producers continue.
- Exhaust a bounded resource.
- Interrupt a multi-step operation halfway through.
- Restore the dependency and observe recovery.
The purpose is not theatrical chaos. It is verifying that the system behaves as its design claims.
Resilience is controlled imperfection
A resilient system is not one that never fails. That goal is unrealistic.
It is a system that fails within understood boundaries, preserves important data, protects critical capabilities, communicates its condition, and recovers without creating a second incident.
Timeouts, retries, circuit breakers, bulkheads, and queues are useful tools. Their presence alone proves very little.
Resilience comes from connecting those mechanisms to business semantics: knowing which work may be repeated, which features may degrade, which state must be reconciled, and which failures must remain isolated.
Partial failure is unavoidable. A system-wide failure is often a design choice.