Interview Question

"Design an end-to-end MLOps pipeline for a production ML system. How do you handle versioning, monitoring, and continuous training? What are the key differences between MLOps and traditional DevOps?"

Difficulty: Hard | Frequently asked at Google, Meta, Amazon

Theoretical Foundation

What is MLOps?

MLOps combines Machine Learning, DevOps, and Data Engineering to operationalize ML systems. It covers the entire lifecycle from data preparation to model monitoring.

Key Components of MLOps

1. Data Management: Versioning, validation, lineage
2. Model Development: Experiment tracking, hyperparameter tuning
3. Model Deployment: Serving, A/B testing, rollback
4. Monitoring: Performance, drift, alerts
5. Governance: Compliance, audit trails, explainability

MLOps vs Traditional DevOps

ML Pipeline Components

Data Pipeline

1. Ingestion: Collect data from sources
2. Validation: Check schema, quality, drift
3. Preprocessing: Transform, normalize, feature engineering
4. Splitting: Train/validation/test sets

Training Pipeline

1. Feature Store: Centralized feature management
2. Training: Distributed training, hyperparameter tuning
3. Evaluation: Model metrics, fairness checks
4. Registry: Version models, metadata, lineage

Deployment Pipeline

1. Serving: Batch, real-time, edge deployment
2. Traffic Management: Canary, blue-green, shadow
3. A/B Testing: Statistical comparison
4. Rollback: Quick recovery from issues

Monitoring Pipeline

1. Performance Monitoring: Accuracy, latency, throughput
2. Data Monitoring: Drift detection, quality checks
3. Business Monitoring: ROI, conversion rates
4. Alerting: Automated notifications

Feature Store

Purpose: Centralized feature management for ML.

Benefits:

• Feature reuse across teams
• Consistent training/serving features
• Feature versioning and lineage
• Low-latency serving

Examples: Feast, Tecton, Hopsworks

Model Registry

Purpose: Version control for models.

Metadata:

• Model version and artifact location
• Training data version
• Hyperparameters and metrics
• Lineage and dependencies
• Approval status

Examples: MLflow, SageMaker Model Registry

CI/CD for ML

Continuous Integration:

• Code testing
• Data validation
• Model training tests
• Integration tests

Continuous Deployment:

• Model serving deployment
• A/B test configuration
• Rollback procedures
• Monitoring setup

Continuous Training:

• Scheduled retraining
• Triggered retraining (drift)
• Feature store updates
• Model performance validation

Monitoring and Observability

Key Metrics to Monitor:

1. Model Metrics:

   • Accuracy, precision, recall, F1
   • AUC-ROC, AUC-PR
   • Calibration error

2. System Metrics:

   • Latency (p50, p95, p99)
   • Throughput (QPS)
   • Error rates

3. Data Metrics:

   • Feature distributions
   • Missing value rates
   • Data drift (PSI, KS test)

4. Business Metrics:

   • Conversion rates
   • Revenue impact
   • User satisfaction

Drift Detection Strategies

1. Statistical Tests: KS test, Chi-squared, PSI
2. Model-based: Train a classifier to distinguish old vs new data
3. Performance-based: Monitor prediction accuracy
4. Automated Retraining: Trigger on drift detection

Code Implementation

Real-World Applications

Google: ML Platform

• Vertex AI: End-to-end ML platform
• TFX: ML pipeline framework
• TF Serving: Model serving at scale
• Continuous training: Automated retraining

Meta: Production ML

• FBLearner: Internal ML platform
• Online learning: Real-time model updates
• A/B testing: Large-scale experimentation
• Model governance: Compliance and audit

Common Follow-Up Questions

Q1: What are the key differences between batch and real-time ML systems? Batch systems process data periodically (hourly/daily) with higher latency tolerance. Real-time systems require sub-second latency and handle streaming data.

Q2: How do you handle feature engineering in production? Use a feature store to ensure consistency between training and serving. Compute features in real-time or pre-compute and store for fast lookup.

Q3: What is shadow deployment and when should you use it? Shadow deployment runs the new model alongside production but doesn't serve predictions. Use it to validate model behavior without affecting users.

Q4: How do you ensure reproducibility in ML pipelines? Version everything: code (Git), data (DVC), models (MLflow), environment (Docker). Use deterministic random seeds and fixed dependencies.

Related Topics

• Model Deployment Strategies
• ML Pipeline Design
• Model Monitoring
• Production ML Systems