Azure VNet, Private Endpoints & Networking

Securing data engineering workloads with private networking, endpoints, and network isolation

Network Architecture for Data Engineering

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    DATA ENGINEERING NETWORK ARCHITECTURE             │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ON-PREMISES              EXPRESS ROUTE                              │
│  ┌──────────┐            ┌──────────┐                               │
│  │ Corporate│◄──────────►│ Express  │                               │
│  │ Network  │            │ Route    │                               │
│  └──────────┘            └────┬─────┘                               │
│                               │                                      │
│                               ▼                                      │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    HUB VNet (10.0.0.0/16)                    │   │
│  │                                                               │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │   │
│  │  │   Gateway    │  │  Firewall    │  │  Bastion     │      │   │
│  │  │   Subnet     │  │  Subnet      │  │  Subnet      │      │   │
│  │  │ 10.0.1.0/24  │  │ 10.0.2.0/24  │  │ 10.0.3.0/24  │      │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘      │   │
│  └─────────────────────────────┬───────────────────────────────┘   │
│                                │                                    │
│                    VNet Peering │ (Transitive)                      │
│                                │                                    │
│  ┌─────────────────────────────▼───────────────────────────────┐   │
│  │                    SPOKE VNet (10.1.0.0/16)                  │   │
│  │                                                               │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │   │
│  │  │ Compute      │  │ Data         │  │ Monitoring   │      │   │
│  │  │ Subnet       │  │ Subnet       │  │ Subnet       │      │   │
│  │  │ 10.1.1.0/24  │  │ 10.1.2.0/24  │  │ 10.1.3.0/24  │      │   │
│  │  │              │  │              │  │              │      │   │
│  │  │ Databricks   │  │ Private End  │  │ Log Analytics│      │   │
│  │  │ ADF IR       │  │ Points       │  │ Monitor      │      │   │
│  │  │ Synapse      │  │              │  │              │      │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘      │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  PRIVATE ENDPOINTS (Private Link):                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐         │
│  │ ADLS PE  │  │ Synapse  │  │ Key Vault │  │ Cosmos   │         │
│  │          │  │ PE       │  │ PE       │  │ DB PE    │         │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘         │
└─────────────────────────────────────────────────────────────────────┘

Private Endpoints Configuration

Bicep Template

// Private DNS Zone for ADLS
resource privateDnsZone 'Microsoft.Network/privateDnsZones@2024-01-01' = {
  name: 'privatelink.dfs.core.windows.net'
  location: 'global'
  properties: {
    maxNumberOfRecordSets: 25000
  }
}

// VNet Link for Private DNS Zone
resource vnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2024-01-01' = {
  name: 'vnet-link-spoke'
  parent: privateDnsZone
  location: 'global'
  properties: {
    virtualNetwork: {
      id: spokeVnet.id
    }
    registrationEnabled: false
  }
}

// Private Endpoint for ADLS Gen2
resource privateEndpoint 'Microsoft.Network/privateEndpoints@2024-01-01' = {
  name: 'pe-adls-gen2'
  location: location
  properties: {
    subnet: {
      id: spokeVnet.properties.subnets[1].id // Data subnet
    }
    privateLinkServiceConnections: [
      {
        name: 'adls-connection'
        properties: {
          privateLinkServiceId: storageAccount.id
          groupIds: ['dfs']
        }
      }
    ]
  }
}

// DNS Zone Group
resource dnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-01-01' = {
  name: 'default'
  parent: privateEndpoint
  properties: {
    privateDnsZoneConfigs: [
      {
        name: 'dnsZoneConfig'
        properties: {
          privateDnsZoneId: privateDnsZone.id
        }
      }
    ]
  }
}

Network Security Groups Rules

{
  "securityRules": [
    {
      "name": "AllowDatabricksInbound",
      "properties": {
        "priority": 100,
        "direction": "Inbound",
        "access": "Allow",
        "protocol": "Tcp",
        "sourcePortRange": "*",
        "destinationPortRange": "443",
        "sourceAddressPrefix": "AzureDatabricks",
        "destinationAddressPrefix": "*"
      }
    },
    {
      "name": "AllowSynapseManagement",
      "properties": {
        "priority": 200,
        "direction": "Inbound",
        "access": "Allow",
        "protocol": "Tcp",
        "sourcePortRange": "*",
        "destinationPortRanges": ["1433", "1443"],
        "sourceAddressPrefix": "SqlManagement",
        "destinationAddressPrefix": "*"
      }
    },
    {
      "name": "DenyAllInbound",
      "properties": {
        "priority": 4096,
        "direction": "Inbound",
        "access": "Deny",
        "protocol": "*",
        "sourcePortRange": "*",
        "destinationPortRange": "*",
        "sourceAddressPrefix": "*",
        "destinationAddressPrefix": "*"
      }
    }
  ]
}

⚠️

Security Critical: Always implement a default Deny rule at the lowest priority (4096) in your NSGs. Never rely solely on Azure service tags for network security—combine with Private Endpoints.

Service Endpoints vs Private Endpoints

Feature	Service Endpoints	Private Endpoints
Traffic Path	Over Azure Backbone	Through Private Link
IP Address	Public IP of service	Private IP in VNet
DNS Resolution	Public DNS	Private DNS Zone
Firewall Support	Service-level only	Full NSG support
Cost	Free	Per endpoint + data
Use Case	Simple scenarios	Enterprise security

VNet Peering Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    VNET PEERING TOPOLOGY                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Region: East US                                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Hub VNet (10.0.0.0/16)                                   │   │
│  │ ┌──────────┐  ┌──────────┐  ┌──────────┐              │   │
│  │ │ Gateway  │  │ Firewall │  │ Bastion  │              │   │
│  │ └──────────┘  └──────────┘  └──────────┘              │   │
│  └────────┬──────────────────┬──────────────────┬─────────┘   │
│           │                  │                  │              │
│           ▼                  ▼                  ▼              │
│  ┌────────────────┐ ┌────────────────┐ ┌────────────────┐    │
│  │ Spoke: DE VNet │ │ Spoke: ML VNet │ │ Spoke: QA VNet │    │
│  │ (10.1.0.0/16)  │ │ (10.2.0.0/16)  │ │ (10.3.0.0/16)  │    │
│  │                 │ │                 │ │                 │    │
│  │ Databricks      │ │ Azure ML        │ │ Dev/Test        │    │
│  │ Synapse         │ │ AML Compute     │ │ Resources       │    │
│  │ ADF              │ │                 │ │                 │    │
│  └────────────────┘ └────────────────┘ └────────────────┘    │
│                                                                 │
│  Region: West US 2                                             │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Hub VNet (10.4.0.0/16) - DR Region                       │   │
│  │ ┌──────────┐  ┌──────────┐                              │   │
│  │ │ Gateway  │  │ Firewall │                              │   │
│  │ └──────────┘  └──────────┘                              │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  GLOBAL VNET PEERING: East US Hub ◄──────────► West US Hub     │
│  (No bandwidth charges within same region peering)              │
│  (Cross-region peering: $0.01/GB ingress + $0.01/GB egress)   │
└─────────────────────────────────────────────────────────────────┘

Azure Data Factory with VNet

{
  "name": "adf-prod-vnet",
  "type": "Microsoft.DataFactory/factories",
  "apiVersion": "2018-06-01",
  "location": "eastus2",
  "identity": {
    "type": "SystemAssigned"
  },
  "properties": {
    "provisioningState": "Succeeded",
    "publicNetworkAccess": "Disabled",
    "networkAcls": {
      "defaultAction": "Deny"
    }
  }
}

Self-Hosted Integration Runtime in VNet

{
  "name": "ir-selfhosted-prod",
  "type": "Microsoft.DataFactory/factories/integrationRuntimes",
  "apiVersion": "2018-06-01",
  "properties": {
    "type": "SelfHosted",
    "typeProperties": {
      "linkedInfo": {
        "type": "LinkedIntegrationRuntimeKey",
        "key": "<EncryptedKey>"
      }
    },
    "hostCaching": "Enabled"
  }
}

ℹ️

Pro Tip: Use Managed Virtual Network with ADF Managed Private Endpoints instead of Self-Hosted IR when possible. It simplifies network management and provides automatic scaling without VM maintenance.

DNS Configuration for Private Endpoints

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    DNS RESOLUTION FLOW                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  CLIENT REQUEST                                                 │
│  ┌──────────┐                                                  │
│  │ ADF IR   │──── stdatalake001.dfs.core.windows.net          │
│  └──────────┘                                                  │
│       │                                                         │
│       ▼                                                         │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Azure DNS (168.63.129.16)                                 │  │
│  │                                                           │  │
│  │ Query: privatelink.dfs.core.windows.net                   │  │
│  │                                                           │  │
│  │ Response: 10.1.2.5 (Private IP from PE)                   │  │
│  └──────────────────────────────────────────────────────────┘  │
│       │                                                         │
│       ▼                                                         │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Private Endpoint (10.1.2.5)                               │  │
│  │                                                           │  │
│  │ ┌──────────────────────────────────────────────────────┐ │  │
│  │ │ Network Interface Card (Private IP)                   │ │  │
│  │ │ Connection to Storage Account via Private Link        │ │  │
│  │ └──────────────────────────────────────────────────────┘ │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
│  DNS ZONES REQUIRED:                                            │
│  • privatelink.dfs.core.windows.net (ADLS Gen2)                │
│  • privatelink.sql.azuresynapse.net (Synapse SQL)              │
│  • privatelink.azuresynapse.net (Synapse Management)           │
│  • privatelink.vaultcore.azure.net (Key Vault)                 │
│  • privatelink.blob.core.windows.net (Blob Storage)            │
└─────────────────────────────────────────────────────────────────┘

Network Security Best Practices

# Python script to audit NSG rules
from azure.identity import DefaultAzureCredential
from azure.mgmt.network import NetworkManagementClient

credential = DefaultAzureCredential()
network_client = NetworkManagementClient(credential, subscription_id)

# List all NSGs and their rules
nsgs = network_client.network_security_groups.list_all()

for nsg in nsgs:
    print(f"\nNSG: {nsg.name}")
    print(f"Resource Group: {nsg.id.split('/')[4]}")
    
    for rule in nsg.security_rules:
        if rule.direction == 'Inbound' and rule.access == 'Allow':
            print(f"  ⚠️  Allow Rule: {rule.name}")
            print(f"     Priority: {rule.priority}")
            print(f"     Source: {rule.source_address_prefix}")
            print(f"     Dest Port: {rule.destination_port_range}")

Interview Questions

Q1: Explain the Hub-Spoke network topology for data engineering. A: The Hub VNet contains shared services (firewall, VPN gateway, Bastion). Spoke VNets contain data engineering workloads (Databricks, Synapse, ADF). Hub-Spoke peering allows centralized management while maintaining network isolation between workloads.

Q2: When would you use Self-Hosted Integration Runtime vs Managed VNet in ADF? A: Self-Hosted IR is needed when connecting to on-premises data sources or when you need custom network routing. Managed VNet with Managed Private Endpoints is preferred for cloud-only scenarios as it eliminates VM management.

Q3: How do Private Endpoints affect data transfer costs? A: Private Endpoints route traffic through Azure Private Link, which avoids public internet traversal. However, data transfer within the same region via Private Endpoints still incurs standard Azure data transfer costs. Cross-region transfers via Private Link incur additional charges.