Azure Cloud Overview & Global Infrastructure

Understanding Microsoft Azure's global footprint, resource management, and foundational services for data engineering

Azure Global Infrastructure

Azure operates the second-largest cloud infrastructure globally with 60+ announced regions spanning 140+ countries. As a data engineer, understanding this infrastructure is critical for designing high-availability, low-latency data solutions.

Regions and Availability Zones

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    AZURE GLOBAL INFRASTRUCTURE                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐     │
│  │   Region     │    │   Region     │    │   Region     │     │
│  │ East US 2    │    │ West Europe  │    │ Southeast A  │     │
│  │              │    │              │    │              │     │
│  │ ┌──────────┐│    │ ┌──────────┐│    │ ┌──────────┐│     │
│  │ │   AZ-1   ││    │ │   AZ-1   ││    │ │   AZ-1   ││     │
│  │ │ ┌──────┐ ││    │ │ ┌──────┐ ││    │ │ ┌──────┐ ││     │
│  │ │ │DC/FC │ ││    │ │ │DC/FC │ ││    │ │ │DC/FC │ ││     │
│  │ │ └──────┘ ││    │ │ └──────┘ ││    │ │ └──────┘ ││     │
│  │ └──────────┘│    │ └──────────┘│    │ └──────────┘│     │
│  │ ┌──────────┐│    │ ┌──────────┐│    │ ┌──────────┐│     │
│  │ │   AZ-2   ││    │ │   AZ-2   ││    │ │   AZ-2   ││     │
│  │ │ ┌──────┐ ││    │ │ ┌──────┐ ││    │ │ ┌──────┐ ││     │
│  │ │ │DC/FC │ ││    │ │ │DC/FC │ ││    │ │ │DC/FC │ ││     │
│  │ │ └──────┘ ││    │ │ └──────┘ ││    │ │ └──────┘ ││     │
│  │ └──────────┘│    │ └──────────┘│    │ └──────────┘│     │
│  │ ┌──────────┐│    │ ┌──────────┐│    │ ┌──────────┐│     │
│  │ │   AZ-3   ││    │ │   AZ-3   ││    │ │   AZ-3   ││     │
│  │ │ ┌──────┐ ││    │ │ ┌──────┐ ││    │ │ ┌──────┐ ││     │
│  │ │ │DC/FC │ ││    │ │ │DC/FC │ ││    │ │ │DC/FC │ ││     │
│  │ │ └──────┘ ││    │ │ └──────┘ ││    │ │ └──────┘ ││     │
│  │ └──────────┘│    │ └──────────┘│    │ └──────────┘│     │
│  └──────────────┘    └──────────────┘    └──────────────┘     │
│                                                                 │
│  DC = Data Center  FC = Floor Controller  AZ = Availability Zone│
│  Each AZ has independent power, cooling, networking            │
└─────────────────────────────────────────────────────────────────┘

Key Concepts

Concept	Description	Data Engineering Impact
Region	Geographic area with 1-3+ AZs	Data residency, latency optimization
Availability Zone	Physically separate datacenter	HA for Synapse, Databricks clusters
Region Pair	Paired regions for DR	Geo-redundant backup of ADLS
Resource Group	Logical container for resources	Organize data engineering assets
Subscription	Billing and access boundary	Cost allocation per project
Management Group	Policy hierarchy	Enterprise governance

ℹ️

Pro Tip: When designing data pipelines, always place your compute (ADF Integration Runtime, Databricks) in the same region as your data storage (ADLS, Synapse) to avoid data transfer costs and latency.

Resource Hierarchy

Architecture Diagram

Management Group (Enterprise)
├── Subscription: Data Engineering Dev
│   ├── Resource Group: rg-datalake-dev
│   │   ├── Storage Account: stdatalake001
│   │   └── Key Vault: kv-secrets-dev
│   └── Resource Group: rg-synapse-dev
│       ├── Synapse Workspace: syn-workspace-dev
│       └── Synapse Managed Vnet
├── Subscription: Data Engineering Prod
│   ├── Resource Group: rg-datalake-prod
│   │   ├── Storage Account: stdatalake001
│   │   └── Key Vault: kv-secrets-prod
│   └── Resource Group: rg-synapse-prod
│       ├── Synapse Workspace: syn-workspace-prod
│       └── Synapse Managed Vnet

ARM Template Example

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountName": {
      "type": "string",
      "metadata": { "description": "ADLS Gen2 storage account name" }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    }
  },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2023-01-01",
      "name": "[parameters('storageAccountName')]",
      "location": "[parameters('location')]",
      "sku": { "name": "Standard_LRS", "tier": "Standard" },
      "kind": "StorageV2",
      "properties": {
        "isHnsEnabled": true,
        "supportsHttpsTrafficOnly": true,
        "minimumTlsVersion": "TLS1_2",
        "accessTier": "Hot",
        "encryption": {
          "services": {
            "blob": { "enabled": true },
            "file": { "enabled": true }
          },
          "keySource": "Microsoft.Storage"
        },
        "networkAcls": {
          "defaultAction": "Deny",
          "virtualNetworkRules": [],
          "ipRules": []
        }
      },
      "tags": {
        "Environment": "Production",
        "Project": "DataEngineering"
      }
    }
  ],
  "outputs": {
    "storageAccountId": {
      "type": "string",
      "value": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
    }
  }
}

Core Services for Data Engineering

Compute Services Comparison

Service	Use Case	Pricing Model	Best For
Azure Functions	Event-driven ETL	Per execution	Lightweight transformations
Azure Data Factory	Orchestration	Per activity run	Complex ETL/ELT workflows
Azure Databricks	Big data processing	Per DBU	Spark-based transformations
Synapse Serverless	Ad-hoc queries	Per TB scanned	Lake exploration
Synapse Dedicated	Reserved compute	Per DWU	Data warehousing
Azure ML	ML pipelines	Per compute	Feature engineering

Storage Services Comparison

Service	Throughput	Latency	Use Case
ADLS Gen2	High	Low	Data lake, analytics
Blob Storage	Very High	Very Low	Object storage, media
Cosmos DB	Very High	Single-digit ms	NoSQL, real-time
Azure Files	Moderate	Low	Shared file systems
Azure NetApp Files	Ultra High	Sub-ms	HPC, SAP HANA

Azure Data Engineering Architecture Pattern

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    TYPICAL DATA ENGINEERING ARCHITECTURE             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  DATA SOURCES          INGESTION           PROCESSING               │
│  ┌──────────┐         ┌──────────┐        ┌──────────┐            │
│  │ On-Prem  │────┬───>│   ADF    │───────>│ Synapse  │            │
│  │ Database │    │    │          │        │ Serverless│            │
│  └──────────┘    │    └──────────┘        └────┬─────┘            │
│                  │                             │                    │
│  ┌──────────┐    │    ┌──────────┐        ┌────▼─────┐            │
│  │ REST API │────┼───>│ Event    │───────>│ Synapse  │            │
│  │          │    │    │ Hubs     │        │ Dedicated│            │
│  └──────────┘    │    └──────────┘        └────┬─────┘            │
│                  │                             │                    │
│  ┌──────────┐    │    ┌──────────┐        ┌────▼─────┐            │
│  │  IoT     │────┴───>│ Stream   │───────>│ Cosmos   │            │
│  │ Devices  │         │Analytics │        │   DB     │            │
│  └──────────┘         └──────────┘        └──────────┘            │
│                                                                     │
│  STORAGE              GOVERNANCE           SERVING                  │
│  ┌──────────┐         ┌──────────┐        ┌──────────┐            │
│  │ ADLS     │<───────>│ Purview  │───────>│ Power BI │            │
│  │ Gen2     │         │          │        │          │            │
│  └──────────┘         └──────────┘        └──────────┘            │
│       │                                                       │    │
│       │              ┌──────────┐        ┌──────────┐         │    │
│       └─────────────>│Key Vault │<───────│ Azure AD │<────────┘    │
│                      └──────────┘        └──────────┘              │
└─────────────────────────────────────────────────────────────────────┘

Azure CLI for Data Engineering Setup

#!/bin/bash

# Create Resource Group
az group create \
  --name "rg-dataengineering-prod" \
  --location "eastus2" \
  --tags Environment=Production Project=DataEngineering

# Create Storage Account with HNS (ADLS Gen2)
az storage account create \
  --name "stdatalakeprodeastus2" \
  --resource-group "rg-dataengineering-prod" \
  --location "eastus2" \
  --sku Standard_LRS \
  --kind StorageV2 \
  --enable-hierarchical-namespace true \
  --min-tls-version TLS1_2 \
  --allow-blob-public-access false \
  --https-only true

# Create containers for data lake zones
az storage container create \
  --name "raw" \
  --account-name "stdatalakeprodeastus2"

az storage container create \
  --name "curated" \
  --account-name "stdatalakeprodeastus2"

az storage container create \
  --name "sandbox" \
  --account-name "stdatalakeprodeastus2"

# Create Synapse Workspace
az synapse workspace create \
  --name "syn-prod-workspace" \
  --resource-group "rg-dataengineering-prod" \
  --location "eastus2" \
  --storage-account "stdatalakeprodeastus2" \
  --file-system "synapsefs" \
  --sql-admin-login-user "sqladmin" \
  --sql-admin-login-password "YourPassword123!"

# Create Synapse SQL Pool (Dedicated)
az synapse sql pool create \
  --name "SQLPool01" \
  --workspace-name "syn-prod-workspace" \
  --resource-group "rg-dataengineering-prod" \
  --performance-level DW100c

⚠️

Important: Always enable HTTPS-only access and TLS 1.2 minimum for all storage accounts. Disable public blob access to prevent data leaks. Use Managed Identities instead of connection strings.

SLA and Performance Guarantees

Service	SLA	RPO	RTO
ADLS Gen2 (RA-GRS)	99.99%	<15 min	<30 min
Synapse Dedicated Pool	99.9%	Point-in-time restore	Hours
Azure Functions	99.95%	N/A	Seconds
Event Hubs	99.95%	0 (with capture)	Minutes
Cosmos DB (Multi-region)	99.999%	0	0
Databricks	99.9%	N/A	Minutes

Pricing Tiers Overview

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    AZURE PRICING MODELS                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  PAY-AS-YOU-GO          RESERVED            SPOT/DEV TEST      │
│  ┌──────────────┐      ┌──────────────┐    ┌──────────────┐   │
│  │ Best for:    │      │ Best for:    │    │ Best for:    │   │
│  │ Development  │      │ Production   │    │ Non-prod     │   │
│  │ Testing      │      │ Stable work  │    │ Dev/test     │   │
│  │ Variable     │      │ Predictable  │    │ Batch jobs   │   │
│  │              │      │              │    │              │   │
│  │ Savings: 0%  │      │ Savings: 30- │    │ Savings: 60- │   │
│  │              │      │ 72%          │    │ 90%          │   │
│  └──────────────┘      └──────────────┘    └──────────────┘   │
│                                                                 │
│  HYBRID BENEFIT        COST MANAGEMENT                          │
│  ┌──────────────┐      ┌──────────────┐                       │
│  │ Use existing  │      │ Budgets      │                       │
│  │ Windows/SQL   │      │ Alerts       │                       │
│  │ licenses      │      │ Advisor      │                       │
│  │              │      │ Cost Analysis│                       │
│  │ Savings: 40% │      │              │                       │
│  └──────────────┘      └──────────────┘                       │
└─────────────────────────────────────────────────────────────────┘

Best Practices Summary

Always use Managed Identities instead of storage keys or connection strings
Enable soft delete on storage accounts for accidental deletion protection
Use Private Endpoints to keep traffic off the public internet
Tag all resources consistently for cost management and governance
Use Availability Zones for production workloads requiring high availability
Implement Azure Policy to enforce security standards across subscriptions
Monitor costs using Azure Cost Management and set up budget alerts
Use ARM/Bicep templates for infrastructure as code (IaC) to ensure consistency

Interview Questions

Q1: Explain the difference between Azure Regions and Availability Zones. A: Regions are geographic areas containing multiple datacenters. Availability Zones are physically separate datacenters within a region, each with independent power, cooling, and networking. For data engineering, use Availability Zones for high availability of critical services like Synapse and Databricks clusters.

Q2: Why should you deploy compute and storage in the same Azure region? A: Deploying in the same region eliminates data transfer costs (which can be significant at scale) and minimizes network latency. For example, ADF Integration Runtime in East US reading from ADLS in East US avoids the $0.01/GB transfer fee.

Q3: What is the benefit of using Azure Resource Groups for data engineering projects? A: Resource Groups provide logical organization, simplified access control (RBAC at RG level), cost tracking per project, and easy cleanup of resources when a project is complete.