Architecture
Design Outbox Pattern
The outbox pattern ensures reliable event publishing by writing events to a database outbox table in the same transaction as business data. A separate process polls the outbox and publishes to the message broker.
- Problem β Dual writes (DB + MQ) are not atomic
- Solution β Transactional outbox with CDC polling
- Guarantee β At-least-once delivery with idempotent consumers
The outbox pattern solves the fundamental problem: you can't write to a database and a message broker atomically.
The Dual Write Problem
When a service writes to both a database and a message broker, there's a window where one write succeeds and the other fails. This violates consistency. The outbox pattern eliminates this window.
Outbox Pattern Solution
Outbox Table Schema
Outbox Record
Here,
- =Entity type (order, user, etc.)
- =Created, Updated, Deleted
- =JSON event data
- =Boolean: published to broker
Polling vs CDC
DfChange Data Capture (CDC)
CDC tools (Debezium, Maxwell) read database WAL/binlog and stream changes as events. This is more efficient than polling and provides real-time event delivery. The outbox table's CDC stream becomes the event source.
| Approach | Latency | Resource Usage | Complexity |
|---|---|---|---|
| Polling | 1-5 seconds | High (queries) | Low |
| CDC | < 1 second | Low (log tailing) | Medium |
| CDC + Kafka | < 100ms | Low | High |
Exactly-Once Delivery
The outbox pattern provides at-least-once delivery. For exactly-once semantics, consumers must be idempotent. Use idempotency keys derived from the event ID or aggregate ID.
Practice Exercises
- Design: Implement an outbox pattern for an order service with Debezium CDC.
- Migration: How would you migrate from polling to CDC without downtime?
- Ordering: How do you ensure events are published in order for a single aggregate?
- Scale: Design an outbox system that handles 100K events per second.
Key Takeaways:
- The outbox pattern solves the dual write problem with transactional consistency
- CDC (Debezium) is preferred over polling for low latency and efficiency
- At-least-once delivery requires idempotent consumers for exactly-once semantics
- Outbox table stores events as part of the business transaction
- Published flag tracks which events have been delivered to the broker
What to Learn Next
-> Saga Pattern Distributed transactions.
-> Idempotency Exactly-once semantics.
-> Back Pressure Event flow control.
-> Design WhatsApp Message delivery guarantees.
-> Design Instagram Event-driven feed generation.
-> Strangler Fig Migrating to event-driven architecture.