Azure Event Hubs: Partitions, Consumer Groups & Capture

Real-time event streaming with partitioning, capture, and exactly-once processing guarantees

Event Hubs Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    AZURE EVENT HUBS ARCHITECTURE                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  PRODUCERS                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │ IoT Hub  │  │ Apps     │  │ Logs     │  │ Custom   │         │
│  │ Devices  │  │ Services │  │ Streams  │  │ Producers│         │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘         │
│       │              │              │              │                │
│       └──────────────┼──────────────┼──────────────┘                │
│                      ▼                                              │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    EVENT HUBS NAMESPACE                       │   │
│  │                                                               │   │
│  │  EVENT HUB: sales-events                                     │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Partition 0 │ Partition 1 │ Partition 2 │ ... │ P N │    │   │
│  │  │ ┌─────────┐ │ ┌─────────┐ │ ┌─────────┐     │     │    │   │
│  │  │ │ Offset  │ │ │ Offset  │ │ │ Offset  │     │     │    │   │
│  │  │ │ Events  │ │ │ Events  │ │ │ Events  │     │     │    │   │
│  │  │ └─────────┘ │ └─────────┘ │ └─────────┘     │     │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  THROUGHPUT UNITS / CAPACITY UNITS                            │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ TU: 1 TU = 1 MB/s in, 2 MB/s out, 1000 events/s    │    │   │
│  │  │ CU: 1 CU = 1 MB/s in, 2 MB/s out (Standard/Prem)   │    │   │
│  │  │ Max: 40 TU (Basic/Standard), 100 CU (Premium)       │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  CONSUMERS                                                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │   │
│  │ │ Stream   │  │ Databricks│  │ Function │  │ Event    │   │   │
│  │ │Analytics │  │ (Spark)  │  │ App      │  │ Grid     │   │   │
│  │ └──────────┘  └──────────┘  └──────────┘  └──────────┘   │   │
│  │                                                             │   │
│  │ CONSUMER GROUPS:                                            │   │
│  │ ┌─────────────────────────────────────────────────────┐   │   │
│  │ │ $Default (all consumers share offset)               │   │   │
│  │ │ stream-analytics-cg (dedicated)                      │   │   │
│  │ │ databricks-cg (dedicated)                           │   │   │
│  │ │ function-cg (dedicated)                             │   │   │
│  │ └─────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  CAPTURE (Auto-archiving to ADLS/Blob):                            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ Event Hubs ──> Capture ──> ADLS Gen2 (Avro/Parquet)         │   │
│  │              Time window: 5 min or 1 MB                      │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Partitioning Strategy

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    PARTITIONING STRATEGY                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  KEY-BASED PARTITIONING                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Event: {"deviceId": "sensor-123", "temp": 72.5}         │   │
│  │                                                           │   │
│  │ Partition Key: deviceId                                   │   │
│  │                                                           │   │
│  │ Hash(deviceId) % NumPartitions = Partition Assignment     │   │
│  │                                                           │   │
│  │ Partition 0: sensor-001, sensor-007, sensor-013...       │   │
│  │ Partition 1: sensor-002, sensor-008, sensor-014...       │   │
│  │ Partition 2: sensor-003, sensor-009, sensor-015...       │   │
│  │                                                           │   │
│  │ Guarantees: Ordering within partition                     │   │
│  │ Limitation: Hot partitions if key distribution skewed     │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ROUND ROBIN PARTITIONING                                       │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ No partition key specified                               │   │
│  │ Events distributed evenly across partitions              │   │
│  │                                                           │   │
│  │ Event 1 → Partition 0                                    │   │
│  │ Event 2 → Partition 1                                    │   │
│  │ Event 3 → Partition 2                                    │   │
│  │ Event 4 → Partition 0                                    │   │
│  │                                                           │   │
│  │ Guarantees: Even distribution                             │   │
│  │ Limitation: No ordering guarantee                         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  PARTITION COUNT CONSIDERATIONS:                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Min: 2 partitions (Basic/Standard)                     │   │
│  │ • Max: 32 partitions (Basic/Standard)                    │   │
│  │ • Max: 128 partitions (Premium)                          │   │
│  │ • Throughput: 1 TU/CU per partition                      │   │
│  │ • Recommendation: 2-4x expected consumer count           │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Event Hubs Capture Configuration

{
  "properties": {
    "captureDescription": {
      "enabled": true,
      "encoding": "Avro",
      "destination": {
        "name": "EventHubArchiveImageFormat",
        "properties": {
          "storageAccountResourceId": "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/stdatalake001",
          "blobContainer": "event-hubs-capture",
          "archiveNameFormat": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}",
          "timeWindow": "00:05:00",
          "sizeLimitInBytes": 104857600,
          "emptyWriterBehavior": "DropIfEmpty"
        }
      },
      "skipEmptyArchive": true
    }
  }
}

Python Producer/Consumer

# Producer
from azure.eventhub import EventHubProducerClient, EventData
import json

producer = EventHubProducerClient.from_connection_string(
    conn_str="Endpoint=sb://ns-prod.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;...",
    eventhub_name="sales-events"
)

events = [
    {"deviceId": "sensor-001", "temperature": 72.5, "timestamp": "2024-01-15T10:30:00Z"},
    {"deviceId": "sensor-002", "temperature": 68.3, "timestamp": "2024-01-15T10:30:01Z"}
]

event_batch = producer.create_batch()
for event in events:
    event_batch.add(EventData(json.dumps(event)))

producer.send_batch(event_batch)
producer.close()

# Consumer
from azure.eventhub import EventHubConsumerClient

def on_event(partition_context, event):
    print(f"Received event from partition {partition_context.partition_id}")
    print(f"Event body: {event.body_as_str()}")
    print(f"Sequence number: {event.sequence_number}")
    partition_context.update_checkpoint(event)

client = EventHubConsumerClient.from_connection_string(
    conn_str="Endpoint=sb://ns-prod.servicebus.windows.net/...",
    consumer_group="$Default",
    eventhub_name="sales-events"
)

with client:
    client.receive(
        on_event=on_event,
        starting_position="-1"  # Start from beginning
    )

ℹ️

Pro Tip: Use partition keys that evenly distribute events across partitions. Avoid using timestamps or sequential IDs as partition keys, as they create hot partitions.

Interview Questions

Q1: Explain the difference between Event Hubs and Event Grid. A: Event Hubs is a high-throughput event streaming service (millions of events/sec). Event Grid is a reactive event routing service (smart routing, filtering). Use Event Hubs for data ingestion pipelines; Event Grid for event-driven architectures.

Q2: How do you handle message ordering in Event Hubs? A: Event Hubs guarantees ordering within a partition. Use the same partition key for related events. For global ordering, use a single partition (limits throughput). For most use cases, partition-level ordering is sufficient.

Q3: What is the cost impact of Event Hubs Capture? A: Capture is included in the Event Hub cost (no additional charges). However, storage costs apply for the captured files in ADLS. Capture reduces the need for custom ETL jobs, saving compute costs.