Unity Catalog: Access Control, Lineage & Marketplace

Centralized governance with Unity Catalog for access control, lineage, and data sharing

Unity Catalog Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    UNITY CATALOG ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  METASTORE (One per region)                                         │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                               │   │
│  │  CATALOGS                                                    │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Catalog: dataengineering                              │    │   │
│  │  │ ┌────────────────────────────────────────────┐       │    │   │
│  │  │ │ Schema: raw                                  │       │    │   │
│  │  │ │ ┌────────────────────────────────────────┐ │       │    │   │
│  │  │ │ │ Tables: sales_raw, inventory_raw        │ │       │    │   │
│  │  │ │ │ Volumes: /mnt/raw_files                │ │       │    │   │
│  │  │ │ │ Functions: validate_data               │ │       │    │   │
│  │  │ └────────────────────────────────────────┘ │       │    │   │
│  │  │ └────────────────────────────────────────────┘       │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  ACCESS CONTROL:                                              │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Principal        │ Catalog  │ Schema │ Permissions   │    │   │
│  │  │ ─────────────────────────────────────────────────── │    │   │
│  │  │ Data Engineers   │ dataeng  │ raw    │ SELECT, MODIFY│    │   │
│  │  │ Data Analysts    │ dataeng  │ curated│ SELECT        │    │   │
│  │  │ ML Engineers     │ dataeng  │ feature│ SELECT        │    │   │
│  │  │ Admins           │ dataeng  │ ALL    │ ALL           │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  LINEAGE:                                                     │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Table ──> Notebook ──> Table ──> Dashboard           │    │   │
│  │  │ Auto-captured from Spark SQL and notebooks           │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Access Control Implementation

# Unity Catalog access control
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import catalog

w = WorkspaceClient()

# Grant SELECT on table
w.grants.update(
    full_name="dataengineering.curated.fact_sales",
    principal="data-analysts@company.com",
    privileges=["SELECT"]
)

# Grant MODIFY on schema
w.grants.update(
    full_name="dataengineering.raw",
    principal="data-engineers@company.com",
    privileges=["SELECT", "MODIFY", "CREATE TABLE"]
)

# Revoke access
w.grants.update(
    full_name="dataengineering.curated.fact_sales",
    principal="former-employee@company.com",
    privileges=["SELECT"],
    action="REVOKE"
)

Lineage Tracking

# Get table lineage
lineage = w.table_lineage.get(
    table_name="dataengineering.curated.fact_sales"
)

for upstream in lineage.upstreams:
    print(f"Upstream: {upstream.table_name}")

for downstream in lineage.downstreams:
    print(f"Downstream: {downstream.table_name}")

Data Sharing (Delta Sharing)

# Create share
w.shares.create(
    name="sales_data_share",
    comment="Shared sales data for partners"
)

# Add table to share
w.shares.update(
    name="sales_data_share",
    add=[
        catalog.ShareTableAdd(
            table_name="dataengineering.curated.fact_sales"
        )
    ]
)

# Create recipient
w.recipients.create(
    name="partner_recipient",
    authentication_type="TOKEN",
    sharing_code="partner-code"
)

ℹ️

Pro Tip: Use Unity Catalog for all data access governance in Databricks. It provides column-level masking, row-level security, and automatic lineage tracking.

Interview Questions

Q1: What are the three levels of the Unity Catalog hierarchy? A: Catalog → Schema → Table/View/Volume. Each level supports access control grants, enabling fine-grained permissions management.

Q2: How does Unity Catalog differ from workspace-level access control? A: Workspace-level control is per-workspace. Unity Catalog provides cross-workspace governance with a central metastore, enabling consistent access policies across all workspaces.

Q3: What is the benefit of automatic lineage in Unity Catalog? A: Automatic lineage tracks data flow from source to consumption, enabling impact analysis, debugging, and compliance reporting without manual documentation.