Azure Data Lake Storage Gen2: Hierarchical, ACLs & POSIX

Enterprise data lake with hierarchical namespace, POSIX ACLs, and high-performance analytics capabilities

ADLS Gen2 Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    ADLS GEN2 ARCHITECTURE                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  STORAGE ACCOUNT (StorageV2 with HNS enabled)                       │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                               │   │
│  │  FILE SYSTEM: datalake                                        │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │                                                       │    │   │
│  │  │  raw/                                                │    │   │
│  │  │  ├── 2024/                                           │    │   │
│  │  │  │   ├── 01/                                         │    │   │
│  │  │  │   │   ├── sales/                                  │    │   │
│  │  │  │   │   │   ├── day=2024-01-01/                    │    │   │
│  │  │  │   │   │   │   ├── part-00000.parquet             │    │   │
│  │  │  │   │   │   │   └── part-00001.parquet             │    │   │
│  │  │  │   │   │   └── day=2024-01-02/                    │    │   │
│  │  │  │   │   │       └── part-00000.parquet             │    │   │
│  │  │  │   │   ├── inventory/                              │    │   │
│  │  │  │   │   └── customers/                              │    │   │
│  │  │  │   └── 02/                                         │    │   │
│  │  │  ├── curated/                                        │    │   │
│  │  │  │   ├── dimensions/                                 │    │   │
│  │  │  │   └── facts/                                      │    │   │
│  │  │  ├── sandbox/                                        │    │   │
│  │  │  └── archive/                                        │    │   │
│  │  │                                                       │    │   │
│  │  │  POSIX ACLs:                                          │    │   │
│  │  │  ┌─────────────────────────────────────────────────┐ │    │   │
│  │  │  │ User/Group      │ Access │ Type  │ Scope        │ │    │   │
│  │  │  │ ─────────────────────────────────────────────── │ │    │   │
│  │  │  │ dataeng-group   │ rwx    │ Allow │ default      │ │    │   │
│  │  │  │ analysts-group  │ r-x    │ Allow │ default      │ │    │   │
│  │  │  │ others          │ ---    │ Deny  │ default      │ │    │   │
│  │  │  └─────────────────────────────────────────────────┘ │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

ACL Configuration

# Set ACL on directory
az storage fs access set \
  --path "raw/2024/01" \
  --account-name stdatalake001 \
  --file-system datalake \
  --group "dataeng-group@company.com" \
  --permission rwx

# Set default ACL (inherited by new items)
az storage fs access set \
  --path "raw/2024/01" \
  --account-name stdatalake001 \
  --file-system datalake \
  --acl "default:user::rwx,default:group::r-x,default:other::---"

# Get current ACL
az storage fs access show \
  --path "raw/2024/01" \
  --account-name stdatalake001 \
  --file-system datalake

# Remove ACL entry
az storage fs access remove \
  --path "raw/2024/01" \
  --account-name stdatalake001 \
  --file-system datalake \
  --acl "group::r-x"

POSIX ACL Permission Model

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    POSIX ACL PERMISSION MODEL                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ACL ENTRY TYPES:                                               │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ user::rwx      - Owner permissions                      │   │
│  │ group::r-x     - Group permissions                      │   │
│  │ other::---     - Others permissions                     │   │
│  │                                                              │   │
│  │ user:<name>:rwx   - Named user                           │   │
│  │ group:<name>:r-x  - Named group                          │   │
│  │ mask::rwx         - Maximum effective permissions        │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  PERMISSION EVALUATION:                                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 1. If user is owner → apply user:: permissions          │   │
│  │ 2. If user is in named group → apply group:<name>       │   │
│  │ 3. If user is in primary group → apply group::          │   │
│  │ 4. Otherwise → apply other:: permissions                │   │
│  │ 5. Effective permissions = ACL ∩ mask                    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ACCESS CHECK ALGORITHM:                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ IF is_owner(user):                                       │   │
│  │   effective = (acl_user & mask)                          │   │
│  │ ELIF in_named_group(user, acl_group):                    │   │
│  │   effective = (acl_group & mask)                         │   │
│  │ ELIF in_primary_group(user, group):                      │   │
│  │   effective = (acl_group & mask)                         │   │
│  │ ELSE:                                                    │   │
│  │   effective = acl_other                                  │   │
│  │                                                          │   │
│  │ Return: (effective & requested_permissions) == requested │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Data Lake Architecture Zones

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    ZONE-BASED DATA LAKE ARCHITECTURE             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  INGESTION ZONE (Raw)                                           │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Raw, unprocessed data                                  │   │
│  │ • Original format preserved                              │   │
│  │ • Immutable (write-once, read-many)                      │   │
│  │ • Schema-on-read                                         │   │
│  │ • TTL: 90 days → Archive tier                           │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            │                                      │
│                            ▼ (ETL/ELT)                            │
│  PROCESSING ZONE (Staging)                                        │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Intermediate transformations                           │   │
│  │ • Data quality validation                                 │   │
│  │ • Schema enforcement (schema-on-write)                   │   │
│  │ • Temporary storage (auto-cleanup)                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            │                                      │
│                            ▼ (Transform)                          │
│  CURATED ZONE (Analytics)                                         │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Cleaned, validated data                                │   │
│  │ • Star/snowflake schema                                  │   │
│  │ • Optimized for query performance                        │   │
│  │ • Partitioned by query patterns                          │   │
│  │ • Delta Lake format for ACID transactions                │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            │                                      │
│                            ▼ (Serve)                              │
│  SERVING ZONE (Consumption)                                       │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Materialized views                                     │   │
│  │ • Aggregate tables                                        │   │
│  │ • Power BI datasets                                       │   │
│  │ • ML feature stores                                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  SANDBOX ZONE (Exploration)                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Ad-hoc analysis                                         │   │
│  │ • Data scientist experimentation                          │   │
│  │ • Self-service analytics                                  │   │
│  │ • TTL: 30 days auto-cleanup                              │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Python SDK for ADLS Gen2

from azure.storage.filedatalake import DataLakeServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = DataLakeServiceClient(
    account_url="https://stdatalake001.dfs.core.windows.net",
    credential=credential
)

# Create directory with ACL
file_system = client.get_file_system_client("datalake")
file_system.create_directory(
    "raw/2024/01/sales",
    metadata={"project": "dataengineering", "retention": "90days"}
)

# Upload with metadata
file_client = client.get_file_client("datalake", "raw/2024/01/sales/data.parquet")
with open("local_file.parquet", "rb") as f:
    file_client.upload_data(f, overwrite=True)

# List directories with properties
paths = list(file_system.list_paths(path="raw/", include_deleted=False))
for path in paths:
    print(f"Name: {path.name}")
    print(f"Is Directory: {path.is_directory}")
    print(f"Last Modified: {path.last_modified}")
    print(f"Content Length: {path.content_length}")

Interview Questions

Q1: Explain the difference between POSIX ACLs and Azure RBAC for ADLS Gen2. A: POSIX ACLs provide file/directory-level permissions with fine-grained control (user, group, mask, other). Azure RBAC provides role-based access at the storage account, container, or directory level. Use POSIX for data lake workloads; RBAC for administrative access.

Q2: How do you implement a data lake zone architecture in ADLS Gen2? A: Create separate containers or directories for each zone (raw, curated, sandbox). Use lifecycle management to tier data. Implement ACLs per zone. Use Delta Lake format in curated zone for ACID transactions.

Q3: What are the performance best practices for ADLS Gen2? A: 1) Use hierarchical namespace for directory operations, 2) Avoid too many small files (aim for 1GB+ per file), 3) Use partitioning for query patterns, 4) Enable ADLS Gen2 API for Hadoop compatibility, 5) Use Parallel File System Operations for bulk uploads.