💰 Cost Optimization

Master cost optimization strategies for S3, compute, and data engineering workloads.

Module: AWS Data Engineering • Topic 28 of 65 • Premium Content

Cost Optimization Framework

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    COST OPTIMIZATION FRAMEWORK                               │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  STORAGE COSTS                                                      │    │
│  │                                                                     │    │
│  │  S3 Standard:     $0.023/GB/mo   → Frequently accessed            │    │
│  │  S3 IA:           $0.0125/GB/mo  → Infrequent (30-day min)        │    │
│  │  S3 Glacier:      $0.004/GB/mo   → Archive (90-day min)           │    │
│  │  S3 Deep Archive: $0.00099/GB/mo → Long-term (180-day min)        │    │
│  │                                                                     │    │
│  │  Savings: Use lifecycle policies → up to 95% reduction             │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  COMPUTE COSTS                                                      │    │
│  │                                                                     │    │
│  │  On-Demand:      Full price      → Dev/Test, variable workloads   │    │
│  │  Reserved (1yr):  ~40% off       → Steady-state production        │    │
│  │  Reserved (3yr):  ~60% off       → Long-term infrastructure       │    │
│  │  Spot Instances:  ~90% off       → Fault-tolerant batch jobs      │    │
│  │  Savings Plans:   Up to 72% off  → Flexible usage patterns        │    │
│  │                                                                     │    │
│  │  Serverless: Lambda, Glue, Athena → Pay only for what you use     │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  DATA TRANSFER COSTS                                                │    │
│  │                                                                     │    │
│  │  Inbound:         Free           → Most data ingestion             │    │
│  │  Outbound:        $0.09/GB       → Use VPC endpoints for S3       │    │
│  │  Cross-AZ:        $0.01/GB       → Keep services in same AZ       │    │
│  │  Cross-Region:    $0.02/GB       → Minimize cross-region traffic  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

S3 Lifecycle Policy Example

{
  "Rules": [
    {
      "ID": "OptimizeDataLake",
      "Status": "Enabled",
      "Filter": {"Prefix": "data-lake/"},
      "Transitions": [
        {"Days": 0, "StorageClass": "STANDARD"},
        {"Days": 90, "StorageClass": "STANDARD_IA"},
        {"Days": 180, "StorageClass": "GLACIER"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ]
    }
  ]
}

Spot Instance Strategy

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    SPOT INSTANCE STRATEGY                                     │
│                                                                             │
│  EMR Cluster Cost Comparison (10 nodes, 24/7):                              │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  On-Demand: 10 × $0.52/hr × 730 hrs = $3,796/month               │    │
│  │  Reserved:  10 × $0.31/hr × 730 hrs = $2,263/month (40% off)     │    │
│  │  Spot:      10 × $0.10/hr × 730 hrs = $730/month (81% off)       │    │
│  │                                                                     │    │
│  │  Annual Savings:                                                    │    │
│  │  Reserved vs On-Demand: $18,396/year                              │    │
│  │  Spot vs On-Demand: $36,792/year                                  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  Best Practices:                                                            │
│  • Use Spot for task nodes (fault-tolerant)                                 │
│  • Use Reserved for core nodes (persistent)                                 │
│  • Use On-Demand for master node (critical)                                 │
│  • Set max price to On-Demand price                                         │
│  • Enable graceful decommissioning                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Interview Q&A

Q1: What is the biggest cost driver in data lakes?

Answer: Storage (S3) and compute (EMR/Glue). Use lifecycle policies for storage and Spot instances for compute to reduce costs.

Q2: How do you estimate costs before deployment?

Answer: Use the AWS Pricing Calculator. Input expected storage, compute, and data transfer requirements.

Q3: When should you use Savings Plans vs. Reserved Instances?

Answer: Savings Plans offer flexibility across instance families and regions. Reserved Instances are specific to instance type and AZ.

Summary

Storage: Lifecycle policies for 95% savings on cold data
Compute: Spot for 90% savings, Reserved for 40-60% savings
Transfer: VPC endpoints eliminate NAT Gateway costs
Serverless: Pay-per-use for variable workloads
Monitoring: Cost Explorer and Budgets for visibility

Cost Optimization for Data Engineering

💰 Cost Optimization