Event Hubs Deep Dive: Capture, Partitioning & Ordering

Master Event Hubs with advanced partitioning, Capture, and throughput optimization

Event Hubs Internals

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    EVENT HUBS INTERNALS                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  NAMESPACE LEVEL                                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Throughput Units (TU): Shared across all Event Hubs       │   │
│  │ • Premium: Capacity Units (CU) per Event Hub                │   │
│  │ • Max 40 TU (Standard), 100 CU (Premium)                    │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  PARTITION LEVEL                                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Each partition: ordered append-only log                    │   │
│  │ • Offset: Position within partition                          │   │
│  │ • Sequence Number: Monotonically increasing per partition    │   │
│  │ • Max 32 partitions (Standard), 128 (Premium)                │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  CONSUMER GROUP LEVEL                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Independent offset tracking per consumer group             │   │
│  │ • Max 1000 consumer groups per Event Hub                    │   │
│  │ • $Default: Shared by all consumers                         │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  CAPTURE LEVEL                                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Auto-archive to ADLS Gen2 / Blob Storage                  │   │
│  │ • Time window: 1 min - 15 min                               │   │
│  │ • Size window: 1 MB - 1 GB                                  │   │
│  │ • Format: Avro or Parquet                                   │   │
│  │ • Ordering: Capture files in offset order per partition      │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Capture Configuration

{
  "properties": {
    "captureDescription": {
      "enabled": true,
      "encoding": "Parquet",
      "destination": {
        "properties": {
          "storageAccountResourceId": "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/stdatalake001",
          "blobContainer": "event-hubs-capture",
          "archiveNameFormat": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}",
          "timeWindow": "00:05:00",
          "sizeLimitInBytes": 104857600
        }
      },
      "skipEmptyArchive": true
    }
  }
}

Partitioning Strategy

# Key-based partitioning for ordering
from azure.eventhub import EventHubProducerClient, EventData

producer = EventHubProducerClient.from_connection_string(conn_str)

# Use device_id as partition key for ordering per device
event = EventData(json.dumps({"device_id": "sensor-001", "temp": 72.5}))
options = {"partition_key": "sensor-001"}
producer.send_batch([event], **options)

# Round-robin for even distribution (no ordering guarantee)
producer.send_batch([event])  # No partition key

Throughput Monitoring

# Monitor Event Hub metrics
from azure.monitor import MonitorManagementClient

monitor_client = MonitorManagementClient(credential, subscription_id)

metrics = monitor_client.metrics.list(
    resource_uri=f"/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.EventHub/namespaces/ns/eventhubs/hub",
    metricnames="IncomingBytes,OutgoingBytes,IncomingMessages",
    interval="PT1M"
)

for metric in metrics.value:
    print(f"{metric.name}: {metric.timeseries[0].data[-1].average}")

ℹ️

Pro Tip: Monitor TU/CU utilization. If consistently above 80%, scale up. Use partition keys that evenly distribute events to avoid hot partitions.

Interview Questions

Q1: How do you scale Event Hubs for higher throughput? A: 1) Increase TU/CU, 2) Add partitions, 3) Use partition keys evenly, 4) Scale consumers, 5) Use Premium tier for dedicated resources. Monitor utilization to determine which approach is needed.

Q2: What is the difference between consumer group and partition? A: Partition is a physical ordering unit within an Event Hub. Consumer group is a logical offset tracking unit. Multiple consumer groups can read the same partition independently.

Q3: How do you replay events from a specific point in time? A: Reset consumer group offset to the desired point. Use sequence number or timestamp. For Capture files, read from the appropriate file in ADLS.