Distributed Reliability
What you will learn
By the end of this lesson, you will be able to:
- treat timeouts and retries as one request budget;
- identify operations that are safe to retry;
- design idempotency for ambiguous outcomes;
- bound concurrency, queues, and overload;
- reason about consistency across database and messaging boundaries.
Mental model: failure is often partial and ambiguous
A remote call can fail before the server sees it, while the server processes it, after the server commits it, or while the response returns.
From the caller's timeout alone, you may not know whether the operation happened.
That uncertainty drives idempotency, status lookup, durable messaging, and reconciliation designs.
One end-to-end budget
Suppose an incoming request has two seconds remaining. Giving each of three dependencies a separate two-second timeout violates the caller's budget.
Propagate the caller context and reserve time for local work and response handling. A downstream operation can use a shorter child deadline:
dependencyCtx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()
result, err := client.Fetch(dependencyCtx, id)
Do not create context.Background() for request work. Cancellation and deadlines must continue through the call graph.
Retry only when the failure and operation permit it
Before retrying, ask:
- Is the failure classified as transient?
- Is the operation idempotent or protected by an idempotency mechanism?
- Is there enough deadline budget for another attempt?
- Is the attempt count bounded?
- Will retries amplify an overloaded dependency?
Use exponential backoff with jitter where appropriate so many clients do not retry in lockstep.
Never retry validation, authentication, most permission failures, or permanent not-found results without a specific consistency reason.
Idempotency keys
For a create operation that clients may safely repeat:
- Client generates a high-entropy idempotency key scoped to the operation/account.
- Server stores the key, a request fingerprint, status, and result durably.
- The first request performs the operation atomically with the idempotency record where possible.
- A repeat with the same key and equivalent request returns the stored result.
- A repeat with the same key but different request is rejected.
Define retention, concurrency, in-progress responses, and scope. An in-memory map does not survive restarts or coordinate replicas.
Backpressure and load shedding
Every scarce resource needs a bound:
- concurrent requests;
- queued jobs;
- database connections;
- outbound dependency calls;
- response and request sizes;
- retry attempts.
When capacity is full, delaying every request indefinitely can cause a cascading failure. Rejecting some work quickly with a clear retry policy may preserve useful service for the rest.
Prioritisation must be explicit and fair enough for the product. Do not let health checks or administrative recovery paths queue behind unbounded normal work.
Circuit breakers
A circuit breaker can stop repeated calls to a failing dependency for a period, allowing recovery and protecting local resources.
It adds state and tuning:
- which failures count;
- rolling window and threshold;
- open duration;
- limited probe behaviour;
- per-instance versus shared state;
- interaction with retries and load balancing.
Add one only after timeouts, bounded concurrency, and useful metrics exist. A poorly tuned breaker can reject healthy work.
Database and message consistency
Writing database state and publishing a message are two separate operations. Either can succeed alone.
An outbox pattern writes the state change and event record in one database transaction. A publisher later sends pending events. Consumers should handle duplicate delivery idempotently because publish acknowledgement can also be ambiguous.
Exactly-once effects across arbitrary systems are not obtained by naming a queue setting. Define deduplication keys, durable state transitions, and reconciliation.
Sagas and compensation
Long workflows across services cannot usually hold one database transaction. A saga records steps and uses compensating actions when a later step fails.
Compensation is business logic, not time travel. Refunding a payment does not erase that it was captured, and notifications already sent cannot be unsent.
Design state machines, retry rules, idempotency, and operator recovery explicitly.
Common mistakes
- Retrying at every layer: attempts multiply and overload the dependency.
- Retrying a timed-out create without idempotency: duplicate resources or charges appear.
- Using one timeout copied everywhere: request types and dependencies have different budgets.
- Adding an unbounded queue during overload: latency and memory grow until failure.
- Calling a message “exactly once” without defining the effect: duplicates still reach business state.
- Using a breaker without metrics: operators cannot understand rejected traffic or recovery.
Practice
Design a payment-order workflow where the client may retry after losing the response. Include idempotency, database state, external payment calls, outbox publication, duplicate events, timeout budgets, and reconciliation.
Create order request
client supplies idempotency key
server transaction inserts pending order + idempotency record
Payment step
worker calls provider with stable provider idempotency key
bounded attempts inside job deadline
ambiguous timeout moves order to payment_unknown, not failed
Provider result
transaction updates order state + inserts OrderPaid outbox event
Outbox publisher
publishes with event ID
marks published after acknowledgement
may publish duplicates after ambiguous acknowledgement
Consumer
stores processed event ID with business update atomically
duplicate ID becomes no-op
Reconciliation
queries provider for payment_unknown orders
advances state from authoritative result
The workflow never assumes a timeout means failure. Every repeatable boundary has a stable key, every queue is bounded, and operations expose states that people and automated reconciliation can repair.
Summary
- Remote failures can leave outcomes unknown, not simply successful or failed.
- Propagate one request budget and bound every retry.
- Retry only transient failures for safe or idempotently protected operations.
- Bound work and shed load before resource exhaustion creates a cascade.
- Use durable state, deduplication, outboxes, and reconciliation for cross-system workflows.
Ready to wrap up?
Mark this complete when you can explain the main idea and finish the practice task.