GCS Architecture for Data Engineering

Google Cloud Storage is the foundation of any GCP data lake. It provides infinitely scalable object storage with strong consistency, versioning, and lifecycle management.

Storage Classes Comparison

Dual-Region and Multi-Region

from google.cloud import storage

client = storage.Client()

# Dual-region bucket (recommended for data engineering)
# Provides 99.999999999% availability (11 9s)
bucket = client.bucket("my-data-lake")
bucket.storage_class = "STANDARD"

# Dual-region location
bucket.location = "US-EAST1"  # Pairs with US-EAST4
bucket.location_type = "dual-region"

bucket = client.create_bucket(bucket, exists_ok=True)
print(f"Created dual-region bucket: {bucket.name} in {bucket.location}")

Bucket Configuration for Data Lakes

from google.cloud import storage
from google.cloud.storage import lifecycle

def create_data_lake_bucket(project_id, bucket_name, region):
    """Create a properly configured data lake bucket."""
    client = storage.Client(project=project_id)

    bucket = client.bucket(bucket_name)
    bucket.storage_class = "STANDARD"
    bucket.location = region
    bucket.versioning_enabled = True

    # Lifecycle rules for data lake tiers
    lifecycle_rules = [
        # Move data to Nearline after 30 days
        lifecycle.LifecycleRuleSetStorageClass("NEARLINE", age=30),
        # Move to Coldline after 90 days
        lifecycle.LifecycleRuleSetStorageClass("COLDLINE", age=90),
        # Move to Archive after 365 days
        lifecycle.LifecycleRuleSetStorageClass("ARCHIVE", age=365),
        # Delete versions older than 730 days
        lifecycle.LifecycleRuleDelete(age=730),
        # Delete noncurrent versions after 30 days
        lifecycle.LifecycleRuleDeleteNoncurrentVersion(noncurrent_days=30),
        # Abort incomplete multipart uploads after 7 days
        lifecycle.LifecycleRuleAbortIncompleteMultipartUpload(
            age_since_custom_time=7
        ),
    ]

    bucket.lifecycle_rules = lifecycle_rules

    # Enable Uniform Bucket-Level Access (recommended)
    bucket.iam_configuration.uniform_bucket_level_access_enabled = True

    # Create bucket
    bucket = client.create_bucket(bucket, exists_ok=True)

    print(f"Data lake bucket created: {bucket.name}")
    print(f"Location: {bucket.location}")
    print(f"Versioning: {bucket.versioning_enabled}")

    return bucket

GCS Integration with BigQuery

External Tables

-- Query Parquet files directly from GCS
CREATE EXTERNAL TABLE `project.dataset.external_sales`
WITH CONNECTION `us-central1.my-connection`
OPTIONS (
  format = 'PARQUET',
  uris = ['gs://my-data-lake/sales/2025/**/*.parquet']
);

-- Query the external table
SELECT
  product_category,
  SUM(revenue) as total_revenue,
  COUNT(*) as transaction_count
FROM `project.dataset.external_sales`
WHERE sale_date >= '2025-01-01'
GROUP BY product_category
ORDER BY total_revenue DESC;

BigLake Tables

-- BigLake table with Iceberg format
CREATE TABLE `project.dataset.iceberg_sales`
WITH CONNECTION `us-central1.my-connection`
OPTIONS (
  format = 'ICEBERG',
  uris = ['gs://my-data-lake/iceberg/sales/metadata/v1.metadata.json']
);

-- BigLake table with Delta Lake format
CREATE TABLE `project.dataset.delta_sales`
WITH CONNECTION `us-central1.my-connection`
OPTIONS (
  format = 'DELTA_LAKE',
  uris = ['gs://my-data-lake/delta/sales/_delta_log/']
);

GCS Object Lifecycle Management

# Set lifecycle rules using gcloud
# Move to Nearline after 30 days
gsutil lifecycle set lifecycle.json gs://my-data-lake

# lifecycle.json
{
  "rule": [
    {
      "action": {
        "type": "SetStorageClass",
        "storageClass": "NEARLINE"
      },
      "condition": {
        "age": 30
      }
    },
    {
      "action": {
        "type": "SetStorageClass",
        "storageClass": "COLDLINE"
      },
      "condition": {
        "age": 90
      }
    },
    {
      "action": {
        "type": "SetStorageClass",
        "storageClass": "ARCHIVE"
      },
      "condition": {
        "age": 365
      }
    },
    {
      "action": {
        "type": "Delete"
      },
      "condition": {
        "age": 730,
        "isLive": false
      }
    }
  ]
}

Autoclass: Intelligent Tiering

Autoclass automatically moves objects between storage classes based on access patterns, eliminating the need for manual lifecycle management.

# Enable Autoclass on a bucket
from google.cloud import storage

client = storage.Client()
bucket = client.bucket("my-autoclass-bucket")

# Enable Autoclass
bucket.autoclass_enabled = True
bucket.autoclass_terminal_storage_class = "ARCHIVE"

bucket = client.create_bucket(bucket, exists_ok=True)
print(f"Autoclass enabled on: {bucket.name}")

Performance Optimization

Parallel Composite Uploads

# For large files (>1GB), use parallel composite uploads
from google.cloud import storage
import multiprocessing

def upload_large_file(bucket_name, source_file, destination_blob):
    """Upload large file using parallel composite upload."""
    client = storage.Client()
    bucket = client.bucket(bucket_name)

    blob = bucket.blob(
        destination_blob,
        chunk_size=256 * 1024 * 1024,  # 256MB chunks
        maximum_chunk_size=512 * 1024 * 1024  # 512MB max
    )

    blob.upload_from_filename(
        source_file,
        timeout=3600  # 1 hour timeout for large files
    )

    print(f"Uploaded {source_file} to {destination_blob}")

Resumable Uploads

# Resumable uploads for reliability
def resumable_upload(bucket_name, source_file, destination_blob):
    """Upload with resumable upload for large files."""
    client = storage.Client()
    bucket = client.bucket(bucket_name)

    blob = bucket.blob(destination_blob)

    # Start resumable upload
    blob.upload_from_filename(
        source_file,
        timeout=3600,
        checksum="crc32c"  # Enable integrity checks
    )

    print(f"Resumable upload complete: {destination_blob}")

GCS for Data Pipeline Patterns

Bronze/Silver/Gold Architecture

Security and Access Control

IAM Policies on Buckets

from google.cloud import storage
from google.iam.v1 import policy_pb2

def set_bucket_iam_policy(bucket_name, member, role):
    """Set IAM policy on a GCS bucket."""
    client = storage.Client()
    bucket = client.bucket(bucket_name)

    policy = bucket.get_iam_policy(requested_policy_version=3)

    # Add binding
    binding = policy_pb2.Binding(
        role=role,
        members=[member]
    )
    policy.bindings.append(binding)

    bucket.set_iam_policy(policy)
    print(f"Updated IAM policy for {bucket_name}")

# Grant BigQuery read access to a bucket
set_bucket_iam_policy(
    "my-data-lake",
    "serviceAccount:bq-load@project.iam.gserviceaccount.com",
    "roles/storage.objectViewer"
)

Signed URLs for Temporary Access

from google.cloud import storage
from datetime import timedelta

def generate_signed_url(bucket_name, blob_name, expiration_hours=1):
    """Generate a signed URL for temporary access."""
    client = storage.Client()
    bucket = client.bucket(bucket_name)
    blob = bucket.blob(blob_name)

    url = blob.generate_signed_url(
        version="v4",
        expiration=timedelta(hours=expiration_hours),
        method="GET",
        content_type="application/octet-stream"
    )

    return url

# Generate 1-hour signed URL for data access
signed_url = generate_signed_url("my-data-lake", "sensitive/data.parquet")
print(f"Access URL: {signed_url}")

Cost Optimization Strategies

# Cost comparison for 1TB of data across storage classes
cost_analysis = {
    "standard": {
        "storage_per_month": 1024 * 0.020,  # $20.48
        "retrieval_per_gb": 0.00,
        "retrieval_100gb": 0.00,
        "total_monthly": 20.48
    },
    "nearline": {
        "storage_per_month": 1024 * 0.010,  # $10.24
        "retrieval_per_gb": 0.01,
        "retrieval_100gb": 1.00,
        "total_monthly": 11.24
    },
    "coldline": {
        "storage_per_month": 1024 * 0.004,  # $4.10
        "retrieval_per_gb": 0.02,
        "retrieval_100gb": 2.00,
        "total_monthly": 6.10
    },
    "archive": {
        "storage_per_month": 1024 * 0.0012,  # $1.23
        "retrieval_per_gb": 0.05,
        "retrieval_100gb": 5.00,
        "total_monthly": 6.23
    }
}

Common Interview Questions

Q1: When would you use Nearline vs. Coldline storage?

Answer: Nearline is for data accessed less than once a month (minimum 30-day storage). Coldline is for data accessed less than once a quarter (minimum 90-day storage). The key difference is retrieval cost: Nearline costs $0.01/GB while Coldline costs$0.02/GB. For data engineering pipelines, use Nearline for recent historical data and Coldline for long-term archival.

Q2: How does Autoclass differ from lifecycle policies?

Answer: Autoclass automatically moves objects between storage classes based on actual access patterns, while lifecycle policies use fixed time-based rules. Autoclass is more dynamic and can save costs when access patterns are unpredictable, but it has minimum storage durations per tier and charges retrieval fees. Lifecycle policies are simpler but less responsive to actual usage.

Q3: What is the significance of dual-region in GCS?

Answer: Dual-region provides 99.999999999% availability (11 nines) by storing data in two specific regions. It offers lower latency than multi-region because data is in geographically closer regions. For data engineering, dual-region is recommended for critical data lakes requiring high availability and specific data residency compliance.

Q4: Explain Bronze/Silver/Gold architecture on GCS.

Answer: Bronze layer stores raw, unprocessed data in GCS (Standard/Nearline). Silver layer contains validated, deduplicated, and cleaned data (GCS Standard). Gold layer is business-ready data in BigQuery or materialized views. This architecture provides data lineage, enables reprocessing, and separates concerns between raw data ingestion and analytics-ready data.

Q5: How do you optimize GCS for BigQuery external tables?

Answer: Use columnar formats (Parquet, ORC) for efficient querying. Partition data by date in the directory structure (year/month/day). Cluster frequently filtered columns. Use appropriate compression (Snappy for Parquet, Zstandard for general). Enable predicate pushdown by filtering on partitioned columns. Consider BigLake tables for Iceberg/Delta Lake formats.