🔮 Databricks on AWS

Master Databricks on AWS with Unity Catalog, clusters, and Delta Lake.

Module: AWS Data Engineering • Topic 31 of 65 • Premium Content

Databricks on AWS Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DATABRICKS ON AWS ARCHITECTURE                             │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  CONTROL PLANE (Databricks-managed)                                 │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Workspace UI, REST API, Jobs, Unity Catalog                 │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│                                ▼                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  DATA PLANE (Your AWS Account)                                      │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  VPC                                                          │  │    │
│  │  │  ┌─────────────────────────────────────────────────────────┐  │  │    │
│  │  │  │  Databricks Clusters                                    │  │  │    │
│  │  │  │  ┌──────────┐ ┌──────────┐ ┌──────────┐               │  │  │    │
│  │  │  │  │ Driver   │ │ Worker 1 │ │ Worker N │               │  │  │    │
│  │  │  │  │ Node     │ │          │ │          │               │  │  │    │
│  │  │  │  └──────────┘ └──────────┘ └──────────┘               │  │  │    │
│  │  │  └─────────────────────────────────────────────────────────┘  │  │    │
│  │  │                                                               │  │    │
│  │  │  ┌─────────────────────────────────────────────────────────┐  │  │    │
│  │  │  │  Storage                                                │  │  │    │
│  │  │  │  ┌──────────┐  ┌──────────┐  ┌──────────┐             │  │  │    │
│  │  │  │  │ S3       │  │ DBFS     │  │ Unity    │             │  │  │    │
│  │  │  │  │ (Data)   │  │ (Root)   │  │ Catalog  │             │  │  │    │
│  │  │  │  └──────────┘  └──────────┘  └──────────┘             │  │  │    │
│  │  │  └─────────────────────────────────────────────────────────┘  │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  UNITY CATALOG                                                      │    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐             │    │
│  │  │ Metastore    │  │ Permissions  │  │ Lineage      │             │    │
│  │  │ (Managed)    │  │ (RBAC)       │  │ (Tracking)   │             │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘             │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Delta Lake

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("DeltaLakeExample") \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .getOrCreate()

# Write Delta table
df.write.format("delta").save("/mnt/delta/sales")

# Merge (upsert) - SCD Type 1
deltaTable = DeltaTable.forPath(spark, "/mnt/delta/sales")

deltaTable.alias("target").merge(
    source=df.alias("source"),
    "target.sale_id = source.sale_id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

# Time travel
df_version0 = spark.read.format("delta").option("versionAsOf", 0).load("/mnt/delta/sales")

# Schema evolution
spark.read.format("delta").option("mergeSchema", "true").load("/mnt/delta/sales")

# Optimize
deltaTable.optimize().executeCompaction()
deltaTable.optimize().executeZOrderBy("sale_date")

Cluster Configuration

# Databricks cluster configuration
cluster_config = {
    "cluster_name": "data-engineering-cluster",
    "spark_version": "13.3.x-scala2.12",
    "node_type_id": "i3.2xlarge",
    "num_workers": 4,
    "autotermination_minutes": 30,
    "spark_conf": {
        "spark.databricks.delta.optimizeWrite.enabled": "true",
        "spark.databricks.delta.autoCompact.enabled": "true",
        "spark.sql.shuffle.partitions": "200",
        "spark.sql.adaptive.enabled": "true"
    },
    "aws_attributes": {
        "availability": "SPOT_WITH_FALLBACK",
        "instance_profile_arn": "arn:aws:iam::123456789012:instance-profile/databricks-role",
        "spot_bid_price_percent": "100",
        "first_on_demand": 1,
        "zone_id": "us-east-1a,us-east-1b,us-east-1c"
    }
}

Interview Q&A

Q1: What is Unity Catalog?

Answer: Unity Catalog is Databricks' unified governance solution for data and AI. It provides centralized access control, auditing, lineage, and data discovery across all workspaces.

Q2: What are the benefits of Delta Lake?

Answer: ACID transactions, schema enforcement/evolution, time travel, data quality (constraints), and unified batch/streaming.

Q3: When should you use Databricks vs. EMR?

Answer: Databricks for collaborative data science, ML workloads, and Delta Lake. EMR for large-scale Spark/Hadoop processing with more infrastructure control.

Summary

Unity Catalog: Centralized governance for data and AI
Delta Lake: ACID transactions, time travel, schema evolution
Clusters: Auto-scaling Spark clusters on AWS
DBFS: Distributed file system backed by S3
Integration: Native integration with AWS services (S3, IAM, KMS)