Azure Blob Storage: Tiers, Lifecycle & Hierarchical Namespace

Mastering Azure Blob Storage architecture, access tiers, lifecycle management, and ADLS Gen2 capabilities

Blob Storage Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    AZURE BLOB STORAGE ARCHITECTURE                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STORAGE ACCOUNT                                                    │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ Type: StorageV2 (General Purpose v2)                         │   │
│  │ Redundancy: LRS / ZRS / GRS / RA-GRS / GZRS / RA-GZRS      │   │
│  │ Access Tier: Hot / Cool / Cold / Archive                     │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  CONTAINERS (Flat Namespace - Blob Storage)                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │   │
│  │ │  raw/    │  │ curated/ │  │ sandbox/ │  │ archive/ │   │   │
│  │ │          │  │          │  │          │  │          │   │   │
│  │ │ Blob1    │  │ Blob1    │  │ Blob1    │  │ Blob1    │   │   │
│  │ │ Blob2    │  │ Blob2    │  │ Blob2    │  │ Blob2    │   │   │
│  │ │ Blob3    │  │ Blob3    │  │ Blob3    │  │ Blob3    │   │   │
│  │ └──────────┘  └──────────┘  └──────────┘  └──────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  FILE SYSTEMS (Hierarchical Namespace - ADLS Gen2)                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ ┌──────────────────────────────────────────────────────┐   │   │
│  │ │  datalake/                                            │   │   │
│  │ │  ├── raw/                                            │   │   │
│  │ │  │   ├── 2024/01/                                    │   │   │
│  │ │  │   │   ├── sales_data.parquet                      │   │   │
│  │ │  │   │   └── inventory_data.parquet                  │   │   │
│  │ │  │   └── 2024/02/                                    │   │   │
│  │ │  ├── curated/                                        │   │   │
│  │ │  │   ├── dimensions/                                 │   │   │
│  │ │  │   └── facts/                                      │   │   │
│  │ │  └── sandbox/                                        │   │   │
│  │ │      └── user_analysis/                              │   │   │
│  │ └──────────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Access Tiers Comparison

Tier	Availability	Latency	Min Retention	Cost (per GB/mo)	Use Case
Hot	99.9%	Milliseconds	None	$0.018	Frequently accessed
Cool	99.9%	Milliseconds	30 days	$0.01	Infrequent access
Cold	99.9%	Milliseconds	90 days	$0.0045	Rare access
Archive	99.9%	Hours	180 days	$0.001	Long-term retention

Tier Transition Costs

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    TIER TRANSITION PRICING                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  REHOVING (Moving UP):                                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Archive → Cold:  $0.0220/GB                              │   │
│  │ Cold → Cool:     $0.0100/GB                              │   │
│  │ Cool → Hot:      $0.0000/GB (free)                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  RETRIEVAL (Reading DOWN):                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Hot:    $0.0000/GB (free)                                │   │
│  │ Cool:   $0.0100/GB                                       │   │
│  │ Cold:   $0.0300/GB                                       │   │
│  │ Archive: Standard: $0.0220/GB (1hr),                     │   │
│  │          High: $0.0440/GB (1hr),                         │   │
│  │          Ultra: $0.0460/GB (1hr)                         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  EARLY DELETION:                                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Cool:     Prorated for remaining 30 days                 │   │
│  │ Cold:     Prorated for remaining 90 days                 │   │
│  │ Archive:  Prorated for remaining 180 days                │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Lifecycle Management Policy

{
  "rules": [
    {
      "enabled": true,
      "name": "MoveToCoolAfter30Days",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            }
          },
          "snapshot": {
            "tierToCool": {
              "daysAfterCreationGreaterThan": 30
          }
        },
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["raw/", "curated/"]
        }
      }
    },
    {
      "enabled": true,
      "name": "MoveToArchiveAfter90Days",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 365
            }
          }
        },
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["raw/"]
        }
      }
    }
  ]
}

Python SDK for Blob Operations

from azure.storage.filedatalake import DataLakeServiceClient
from azure.identity import DefaultAzureCredential
import datetime

credential = DefaultAzureCredential()
service_client = DataLakeServiceClient(
    account_url="https://stdatalake001.dfs.core.windows.net",
    credential=credential
)

# Create file system (container)
file_system = service_client.create_file_system(
    file_system_name="raw",
    public_access=None
)

# Upload file with tier
file_client = service_client.get_file_client(
    file_system="raw",
    file_name="2024/01/sales_data.parquet"
)

with open("local_sales_data.parquet", "rb") as data:
    file_client.upload_data(
        data,
        overwrite=True,
        max_concurrency=4
    )

# Set access tier
file_client.set_standard_blob_tier(tier="Cool")

# List files with properties
file_system_client = service_client.get_file_system_client("raw")
paths = list(file_system_client.list_paths(path="2024/01/"))
for path in paths:
    props = path
    print(f"Name: {path.name}")
    print(f"Size: {path.size} bytes")
    print(f"Last Modified: {path.last_modified}")
    print(f"Content Type: {path.content_type}")

ℹ️

Pro Tip: Use AzCopy for large-scale data transfers—it's 10x faster than Azure Portal uploads and supports parallel transfers, tier setting, and lifecycle policy triggers.

AzCopy Commands for Data Engineering

# Copy data from on-prem to ADLS Gen2
azcopy copy "https://onpremserver/data/*.parquet" \
  "https://stdatalake001.dfs.core.windows.net/raw/2024/01/?<SAS>" \
  --recursive \
  --overwrite=true \
  --block-size-mb=256

# Set blob tier during copy
azcopy copy "https://source.blob.core.windows.net/container/*" \
  "https://stdatalake001.blob.core.windows.net/archive/?<SAS>" \
  --blob-type=BlockBlob \
  --block-blob-tier=Archive

# Sync directory (incremental)
azcopy sync "https://stdatalake001.dfs.core.windows.net/raw/2024/01/?<SAS>" \
  "C:\local-backup" \
  --delete-destination=true

# List all blobs with properties
azcopy list "https://stdatalake001.dfs.core.windows.net/raw/?<SAS>" \
  --machine-readable \
  --overwrite=false \
  --output-type=json

Redundancy Options

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    STORAGE REDUNDANCY OPTIONS                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  LRS (Locally Redundant Storage)                               │
│  ┌─────────────────────────────────────────────────────┐       │
│  │ Single Data Center                                     │       │
│  │ ┌──────────┐ ┌──────────┐ ┌──────────┐             │       │
│  │ │ Copy 1   │ │ Copy 2   │ │ Copy 3   │             │       │
│  │ └──────────┘ └──────────┘ └──────────┘             │       │
│  │ Durability: 99.999999999% (11 nines)               │       │
│  └─────────────────────────────────────────────────────┘       │
│                                                                 │
│  ZRS (Zone-Redundant Storage)                                  │
│  ┌─────────────────────────────────────────────────────┐       │
│  │ 3 Availability Zones                                  │       │
│  │ ┌──────────┐ ┌──────────┐ ┌──────────┐             │       │
│  │ │   AZ 1   │ │   AZ 2   │ │   AZ 3   │             │       │
│  │ │ Copy 1   │ │ Copy 2   │ │ Copy 3   │             │       │
│  │ └──────────┘ └──────────┘ └──────────┘             │       │
│  │ Durability: 99.999999999% (11 nines)               │       │
│  └─────────────────────────────────────────────────────┘       │
│                                                                 │
│  GRS (Geo-Redundant Storage)                                   │
│  ┌─────────────────────────────────────────────────────┐       │
│  │ Primary Region              Secondary Region         │       │
│  │ ┌──────────────────┐      ┌──────────────────┐     │       │
│  │ │ ┌──┐ ┌──┐ ┌──┐  │─────>│ ┌──┐ ┌──┐ ┌──┐  │     │       │
│  │ │ │C1│ │C2│ │C3│  │      │ │C1│ │C2│ │C3│  │     │       │
│  │ │ └──┘ └──┘ └──┘  │ Async│ └──┘ └──┘ └──┘  │     │       │
│  │ └──────────────────┘      └──────────────────┘     │       │
│  │ Durability: 99.999999999% (11 nines)               │       │
│  └─────────────────────────────────────────────────────┘       │
│                                                                 │
│  RA-GRS (Read-Access Geo-Redundant Storage)                    │
│  Same as GRS but allows READ access to secondary                │
│  even during primary region outage                              │
└─────────────────────────────────────────────────────────────────┘

Interview Questions

Q1: When would you use Hot vs Cool vs Archive tiers for data engineering? A: Hot for frequently queried data (last 30 days), Cool for infrequent access (30-90 days), Archive for long-term compliance (90+ days). Example: raw data starts Hot, moves to Cool after 30 days, Archive after 90 days.

Q2: What is the impact of enabling Hierarchical Namespace on a storage account? A: HNS enables ADLS Gen2 capabilities: POSIX ACLs, directory operations, atomic renames, and improved Hadoop compatibility. However, it changes the API endpoint from blob to dfs and some Blob Storage features may not be available.

Q3: How do lifecycle management policies affect data engineering costs? A: Lifecycle policies automatically move data to cheaper tiers, reducing storage costs by 50-90%. However, retrieval costs must be considered—frequently accessed archived data can cost more than keeping it in Hot tier.