Theory
Design Idempotency
Idempotency ensures that performing an operation multiple times produces the same result as performing it once. This is critical in distributed systems where retries are inevitable.
- Problem β Network failures cause duplicate requests
- Solution β Idempotency keys for deduplication
- Guarantee β Exactly-once processing semantics
Idempotency is the foundation of reliable distributed systems: if retries are safe, the system is resilient.
What Is Idempotency?
DfIdempotent Operation
An operation f is idempotent if f(f(x)) = f(x). In distributed systems, this means: if a client sends the same request twice, the server processes it only once. The second request returns the cached result, not re-execute the operation.
HTTP methods GET, PUT, DELETE are naturally idempotent. POST and PATCH are not. This is why payment APIs use idempotency keys: a retry of a payment request should not charge the customer twice.
Idempotency Key Design
Idempotency Key
Here,
- =Globally unique identifier
- =Deterministic hash for retry detection
Implementation Pattern
def process_payment(user_id, amount, idempotency_key):
# Check if key exists in Redis
if redis.exists(f"idemp:{idempotency_key}"):
return redis.get(f"idemp:{idempotency_key}")
# Set key with TTL (24 hours)
redis.setex(f"idemp:{idempotency_key}", 86400, "processing")
try:
result = charge_payment(user_id, amount)
redis.setex(f"idemp:{idempotency_key}", 86400, result)
return result
except Exception as e:
redis.delete(f"idemp:{idempotency_key}")
raise
Exactly-Once Semantics
DfExactly-Once Processing
Exactly-once processing means each message is processed exactly one time, never more, never less. This is achieved through: (1) Idempotent consumers, (2) Transactional outbox pattern, (3) Message deduplication at the broker level.
True exactly-once delivery is impossible in distributed systems (Two Generals Problem). Instead, we achieve effectively-once semantics through idempotent operations and deduplication.
Deduplication Window
Deduplication TTL
Here,
- =How long clients retry (e.g., 24h)
- =Max partition duration (e.g., 5min)
Practice Exercises
- Design: Implement an idempotent payment API that handles concurrent duplicate requests.
- Storage: Design a deduplication system for 1M requests/second with 24-hour TTL.
- Migration: How would you add idempotency to an existing non-idempotent API without downtime?
- Consistency: Design a system where idempotency keys are generated client-side vs server-side.
Key Takeaways:
- Idempotency ensures operations produce the same result when repeated
- Idempotency keys stored in Redis with TTL enable deduplication
- HTTP PUT/DELETE are naturally idempotent; POST requires explicit handling
- True exactly-once is impossible; use effectively-once with idempotency
- Deduplication window should cover maximum retry duration
What to Learn Next
-> Saga Pattern Distributed transactions with idempotency.
-> Outbox Pattern Reliable event publishing.
-> Retry Patterns Resilient retry with backoff.
-> Circuit Breaker Preventing cascade failures.
-> Back Pressure Load management.
-> Design Amazon Idempotent checkout.