Azure DevOps: Pipelines, Repos & Artifacts for Data

CI/CD automation with Azure DevOps for data engineering workloads

DevOps for Data Engineering

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    DEVOPS FOR DATA ENGINEERING                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  REPOS                  BUILD                 RELEASE                 │
│  ┌──────────┐         ┌──────────────┐      ┌──────────────┐      │
│  │ ADF Git  │────────>│ Azure DevOps │─────>│ Dev          │      │
│  │ ARM/Bicep│         │ Build        │      │ Environment  │      │
│  │ Python   │         │ Pipeline     │      │              │      │
│  │ SQL      │         │              │      └──────┬───────┘      │
│  └──────────┘         │ • Validate   │             │                │
│                       │ • Package    │             ▼                │
│                       │ • Test       │      ┌──────────────┐      │
│                       └──────────────┘      │ QA           │      │
│                                             │ Environment  │      │
│  ┌──────────────┐                          └──────┬───────┘      │
│  │ Artifacts    │                                 │                │
│  │ (Packages)   │─────────────────────────────────┘                │
│  │              │                                                  │
│  │ • Python     │                          ┌──────────────┐      │
│  │ • NuGet      │─────────────────────────>│ Production   │      │
│  │ • Maven      │                          │ Environment  │      │
│  └──────────────┘                          └──────────────┘      │
│                                                                     │
│  TESTS                                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Unit Tests: Validate transformation logic                  │   │
│  │ • Integration Tests: Test pipeline connectivity              │   │
│  │ • Data Quality Tests: Validate output data                   │   │
│  │ • Performance Tests: Benchmark query performance             │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

YAML Pipeline Example

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - feature/*

variables:
  - group: dataengineering-variables

stages:
  - stage: Build
    jobs:
      - job: ValidateAndTest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.9'

          - script: |
              pip install pytest
              pytest tests/ --junitxml=test-results.xml
            displayName: 'Run Unit Tests'

          - task: PublishTestResults@2
            inputs:
              testResultsFiles: 'test-results.xml'

  - stage: DeployDev
    dependsOn: Build
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployToDev
        environment: 'dev'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: 'dataengineering-dev'
                    scriptType: 'bash'
                    inlineScript: |
                      az deployment group create \
                        --resource-group rg-dataengineering-dev \
                        --template-file infra/main.bicep \
                        --parameters environment=dev

  - stage: DeployProd
    dependsOn: DeployDev
    condition: succeeded()
    jobs:
      - deployment: DeployToProd
        environment: 'prod'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: 'dataengineering-prod'
                    scriptType: 'bash'
                    inlineScript: |
                      az deployment group create \
                        --resource-group rg-dataengineering-prod \
                        --template-file infra/main.bicep \
                        --parameters environment=prod

Branch Strategy

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    GIT BRANCH STRATEGY                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  main (Production)                                              │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Stable, production-ready code                            │   │
│  │ Protected branch: PR required, builds must pass         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                          ▲                                      │
│                          │ PR                                   │
│  develop (Integration)    │                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Integration branch for feature merges                    │   │
│  │ Auto-deploy to Dev environment                           │   │
│  └─────────────────────────────────────────────────────────┘   │
│                          ▲                                      │
│                          │ PR                                   │
│  feature/* (Features)     │                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Individual feature branches                              │   │
│  │ Developers work here                                     │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

ℹ️

Pro Tip: Use branch policies to require PR reviews and successful builds before merging to main. This ensures code quality and prevents broken deployments.

Interview Questions

Q1: How do you implement CI/CD for ADF with Azure DevOps? A: 1) Enable ADF Git integration, 2) Create build pipeline for validation, 3) Use ARM/Bicep templates for infrastructure, 4) Create release pipeline with environment promotion, 5) Implement approval gates for production.

Q2: What is the difference between build and release pipelines? A: Build pipelines compile, test, and package code. Release pipelines deploy artifacts to environments with approval gates and deployment strategies.

Q3: How do you handle rollback in Azure DevOps? A: Use deployment slots for blue-green deployments. Keep previous version artifacts. Implement automated rollback triggers on failure. Use infrastructure as code for environment recreation.