Data Systems
Batch Processing
Batch processing handles large volumes of data efficiently through parallelization. Master MapReduce, Apache Spark, and the fundamentals of distributed data processing.
- Parallelism β Process data in parallel across many machines
- Fault Tolerance β Automatic recovery from failures
- Scalability β Linear performance improvement with more machines
Batch processing turns days of work into hours, and hours into minutes.
Batch Processing Fundamentals
DfBatch Processing
Batch processing is a computing paradigm where a set of jobs is collected and processed together as a single unit. It is optimized for high throughput on large datasets, processing data in parallel across many machines. Batch processing is ideal for ETL pipelines, analytics, and machine learning training.
MapReduce
DfMapReduce
MapReduce is a programming model for processing large datasets in parallel across a distributed cluster. It consists of two phases: Map (transform and filter data) and Reduce (aggregate results). The framework handles data distribution, fault tolerance, and result aggregation automatically.
Apache Spark
DfApache Spark
Apache Spark is a unified analytics engine for large-scale data processing. It provides in-memory computation, making it 10-100x faster than MapReduce for iterative algorithms. Spark supports batch processing, stream processing, machine learning (MLlib), and graph processing (GraphX).
| Component | Purpose |
|---|---|
| Spark Core | RDD abstraction, task scheduling |
| Spark SQL | Structured data processing with DataFrames |
| Spark Streaming | Micro-batch stream processing |
| MLlib | Machine learning library |
| GraphX | Graph processing |
RDD vs DataFrame vs Dataset
| Abstraction | Type Safety | Optimization | Use Case |
|---|---|---|---|
| RDD | Compile-time | None | Low-level control |
| DataFrame | Runtime | Catalyst optimizer | SQL-like queries |
| Dataset | Compile-time | Catalyst optimizer | Type-safe queries |
Prefer DataFrames/Datasets over RDDs. The Catalyst optimizer performs query planning, predicate pushdown, and code generation, resulting in significantly better performance with less code.
Data Parallelism
DfData Parallelism
Data parallelism is the technique of splitting data across multiple workers, each performing the same operation on its partition. This is the fundamental pattern in batch processing: each worker processes a subset of the data independently, and results are combined at the end.
Speedup from Parallelism
Here,
- =Performance improvement factor
- =Time for serial execution
- =Number of parallel workers
- =Coordination and communication overhead
Parallel Word Count
For a 1TB text file with 100 mappers and 10 reducers:
Serial approach: Read 1TB sequentially β hours Parallel approach: 100 mappers each read 10GB β minutes
Each mapper counts words in its split, emits (word, 1) pairs. Shuffler groups by key. Each reducer sums counts for its assigned words.
With 100 mappers, theoretical speedup β 100x (minus overhead).
Fault Tolerance
| Mechanism | Description |
|---|---|
| Data replication | Store input data on multiple nodes |
| Task retry | Re-execute failed tasks on other nodes |
| Checkpointing | Save intermediate state to durable storage |
| Lineage | Rebuild lost data from transformation history |
Spark's RDD lineage enables fault tolerance without data replication. If a partition is lost, Spark recomputes it from the source using the transformation graph. This is efficient for CPU-bound workloads but can be expensive for long lineage chains.
Batch vs Stream: When to Use Each
| Use Case | Recommended | Reason |
|---|---|---|
| Daily analytics report | Batch | Complete data needed |
| Real-time dashboard | Stream | Low latency required |
| ML model training | Batch | Large dataset, no time pressure |
| Fraud detection | Stream | Immediate response needed |
| ETL pipeline | Batch | High throughput, predictable schedule |
| IoT sensor monitoring | Stream | Continuous data, real-time alerts |
Practice Exercises
-
MapReduce Design: Design a MapReduce job to find the top 10 most frequent words in a 10TB text corpus. What are the map and reduce functions?
-
Spark Optimization: Given a Spark job that processes 1TB of data, identify 3 optimization strategies to reduce processing time from 2 hours to 30 minutes.
-
Fault Tolerance: Explain how Spark's lineage-based fault tolerance works. What are the trade-offs compared to HDFS replication?
-
Architecture Decision: Design a batch processing pipeline for daily ETL from PostgreSQL to a data warehouse. What components would you include, and how do you handle failures?
Key Takeaways:
- MapReduce provides a simple model for parallel data processing
- Spark uses in-memory computation for 10-100x speedup over MapReduce
- Data parallelism splits work across workers for horizontal scaling
- Fault tolerance uses checkpointing, replication, or lineage
- Prefer DataFrames/Datasets over RDDs for Catalyst optimization
- Batch processing is ideal for high-throughput, latency-tolerant workloads
What to Learn Next
-> Stream Processing Real-time data processing with Flink, Spark Streaming, and Kafka Streams.
-> Data Lake Architecture Storage, processing, and governance for large-scale data.
-> Kafka Deep Dive Event streaming, partitioning, and exactly-once semantics.
-> Message Queues Async processing, event-driven architecture, and pub/sub patterns.
-> Event-Driven Architecture Event sourcing, CQRS, and message-driven systems.
-> Observability Logging, metrics, tracing, and monitoring distributed systems.