🕸️ Data Mesh on AWS

Implement Data Mesh architecture using Lake Formation and cross-account sharing.

Module: AWS Data Engineering • Topic 22 of 65 • Premium Content

Data Mesh Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DATA MESH ON AWS                                           │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  DOMAIN: Sales                                                      │    │
│  │  Account: 111111111111                                              │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  S3 Bucket: sales-data-lake                                   │  │    │
│  │  │  Glue Catalog: sales_catalog                                  │  │    │
│  │  │  Lake Formation: sales-domain-permissions                     │  │    │
│  │  │  Tables: customers, orders, transactions                      │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │ Share via Lake Formation                   │
│                                ▼                                            │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  CENTRAL GOVERNANCE ACCOUNT (222222222222)                          │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Lake Formation Cross-Account Permissions                     │  │    │
│  │  │  • Register locations                                        │  │    │
│  │  │  • Manage sharing policies                                    │  │    │
│  │  │  • Audit access patterns                                      │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │ Share via Lake Formation                   │
│              ┌─────────────────┼─────────────────┐                         │
│              ▼                 ▼                 ▼                         │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐            │
│  │  Marketing      │  │  Finance        │  │  Analytics      │            │
│  │  (333333333333) │  │  (444444444444) │  │  (555555555555) │            │
│  │  Query sales    │  │  Query sales    │  │  Query all      │            │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘            │
└─────────────────────────────────────────────────────────────────────────────┘

Cross-Account Sharing

import boto3

lakeformation = boto3.client('lakeformation')

# Share table with another account
lakeformation.register_resource(
    ResourceArn='arn:aws:s3:::sales-data-lake/silver/customers/',
    RoleArn='arn:aws:iam::111111111111:role/LakeFormationRole',
    UseServiceLinkedRole=False
)

# Grant cross-account access
lakeformation.batch_grant_permissions(
    Entries=[
        {
            'Id': 'share-customers',
            'Principal': {
                'DataLakePrincipalIdentifier': 'arn:aws:iam::333333333333:role/AnalystRole'
            },
            'Resource': {
                'Table': {
                    'DatabaseName': 'sales_catalog',
                    'Name': 'customers'
                }
            },
            'Permissions': ['SELECT'],
            'GrantOption': False
        }
    ]
)

Interview Q&A

Q1: What is Data Mesh?

Answer: Data Mesh is a decentralized data architecture where domain teams own and serve their data as products, using self-serve infrastructure.

Q2: How does Lake Formation enable Data Mesh?

Answer: Lake Formation provides cross-account sharing, fine-grained permissions, and centralized governance for decentralized data ownership.

Q3: What are the four principles of Data Mesh?

Answer: 1) Domain ownership, 2) Data as a product, 3) Self-serve platform, 4) Federated computational governance.

Summary

Data Mesh: Decentralized, domain-oriented data architecture
Lake Formation: Enables cross-account sharing with fine-grained permissions
Cross-Account: Share tables/columns across AWS accounts
Governance: Central policies with federated execution
Data Products: Domains own and serve their data with quality SLAs