Chapter 2: MLOps vs DevOps vs DataOps
Understanding the Ops Landscape
The modern data and software landscape involves multiple "Ops" disciplines. Understanding how they relate and differ is crucial for building effective teams and systems.
Comparison Overview
Detailed Comparison Table
| Aspect | DevOps | DataOps | MLOps |
|---|---|---|---|
| Primary Focus | Software delivery | Data pipeline delivery | ML model delivery |
| Main Artifact | Application code | Data products | ML models |
| Testing | Unit, integration, E2E | Data quality, schema | Model validation, A/B |
| Versioning | Code only | Data lineage | Code + Data + Models |
| Monitoring | App metrics, logs | Data quality metrics | Model performance, drift |
| CI/CD Target | Deploy applications | Deploy data pipelines | Deploy models |
| Feedback Loop | Bug reports | Data issues | Model degradation |
DevOps: The Foundation
DevOps focuses on shortening the development lifecycle while delivering features, fixes, and updates frequently.
Core DevOps Practices
# Example: Standard DevOps CI/CD Pipeline
name: Application CI/CD
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build
run: npm run build
- name: Unit Tests
run: npm test
- name: Integration Tests
run: npm run test:integration
- name: Build Docker Image
run: docker build -t myapp:${{ github.sha }} .
deploy:
needs: build
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to Production
run: kubectl apply -f k8s/
DevOps Tools
| Category | Tools |
|---|---|
| Version Control | Git, GitHub, GitLab |
| CI/CD | Jenkins, GitHub Actions, GitLab CI |
| Containers | Docker, Podman |
| Orchestration | Kubernetes, Docker Swarm |
| Monitoring | Prometheus, Grafana, Datadog |
| IaC | Terraform, Ansible, Pulumi |
DataOps: Managing Data Pipelines
DataOps applies DevOps principles to data analytics and data engineering workflows.
Core DataOps Practices
# Example: DataOps Pipeline with Great Expectations
import great_expectations as gx
from great_expectations.core.batch import BatchRequest
# Create data context
context = gx.get_context()
# Define expectations for data quality
validator = context.get_validator(
batch_request=BatchRequest(
datasource_name="my_datasource",
data_connector_name="default_inferred_data_connector",
data_asset_name="sales_data",
),
expectation_suite_name="sales_data_expectations"
)
# Data quality checks
validator.expect_column_values_to_not_be_null("customer_id")
validator.expect_column_values_to_be_between("quantity", min_value=1, max_value=1000)
validator.expect_column_values_to_be_unique("order_id")
validator.expect_column_values_to_match_regex("email", r"^[\w\.-]+@[\w\.-]+\.\w+$")
# Save expectations
validator.save_expectation_suite(discard_failed_expectations=False)
DataOps Tools
| Category | Tools |
|---|---|
| Orchestration | Apache Airflow, Dagster, Prefect |
| Data Quality | Great Expectations, dbt tests, Soda |
| Data Catalog | DataHub, Amundsen, Apache Atlas |
| ETL/ELT | dbt, Spark, Fivetran |
| Data Lineage | OpenLineage, Marquez |
MLOps: The Intersection
MLOps sits at the intersection of DevOps and DataOps, inheriting practices from both while adding ML-specific capabilities.
What MLOps Adds
MLOps-Specific Challenges
| Challenge | Description | Solution |
|---|---|---|
| Model Reproducibility | Same code, different results | Seed fixing, environment locking |
| Data Drift | Input data changes over time | Drift detection, monitoring |
| Model Decay | Model performance degrades | Continuous training, alerts |
| Experiment Management | Tracking many experiments | MLflow, W&B, Neptune |
| Feature Consistency | Train/serve feature mismatch | Feature stores |
Team Structures
Option 1: Centralized MLOps Team
┌─────────────────────────────────────────────────────────┐
│ MLOps Platform Team │
├─────────────────────────────────────────────────────────┤
│ • Owns ML infrastructure │
│ • Provides self-service tools │
│ • Sets standards and best practices │
│ • Supports multiple ML teams │
└─────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ ML Team │ │ ML Team │ │ ML Team │
│ A │ │ B │ │ C │
└─────────┘ └─────────┘ └─────────┘
Pros:
- Consistent standards
- Efficient resource utilization
- Specialized expertise
Cons:
- Potential bottleneck
- May not understand specific team needs
Option 2: Embedded MLOps Engineers
┌─────────────────────────────────────────────────────────┐
│ Product Team A │
├─────────────────────────────────────────────────────────┤
│ Data Scientists (3) + ML Engineer (1) + MLOps (1) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Product Team B │
├─────────────────────────────────────────────────────────┤
│ Data Scientists (2) + ML Engineer (2) + MLOps (1) │
└─────────────────────────────────────────────────────────┘
Pros:
- Deep domain knowledge
- Fast iteration
- Direct communication
Cons:
- Duplicated efforts
- Inconsistent practices
- Resource intensive
Option 3: Hybrid Model (Recommended)
┌─────────────────────────────────────────────────────────┐
│ Central MLOps Platform Team │
│ (Infrastructure, Tools, Standards, Support) │
└─────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Product │ │ Product │ │ Product │
│ Team A │ │ Team B │ │ Team C │
│ │ │ │ │ │
│ ML Eng ◄──┼───────┼──► Shared │◄──────┼─► ML Eng │
│ │ │ Tools │ │ │
└───────────┘ └───────────┘ └───────────┘
Responsibilities Matrix
RACI Matrix for ML Project
| Activity | Data Scientist | ML Engineer | MLOps Engineer | Data Engineer |
|---|---|---|---|---|
| Data Collection | C | I | I | R/A |
| Feature Engineering | R/A | C | I | C |
| Model Development | R/A | C | I | I |
| Training Pipeline | C | R/A | C | I |
| Model Deployment | I | C | R/A | I |
| Monitoring | I | C | R/A | C |
| Infrastructure | I | I | R/A | I |
Legend: R = Responsible, A = Accountable, C = Consulted, I = Informed
Collaboration Points
Data Handoff (DataOps → MLOps)
# Example: Data contract between DataOps and MLOps
data_contract:
name: customer_features
version: 1.2.0
owner: data-engineering@company.com
schema:
customer_id:
type: string
nullable: false
pattern: "^CUS-[0-9]{8}$"
age:
type: integer
nullable: false
min: 18
max: 120
total_purchases:
type: float
nullable: false
min: 0.0
sla:
freshness: 1 hour
availability: 99.9%
delivery:
format: parquet
location: s3://data-lake/features/customer/
update_frequency: hourly
Deployment Handoff (MLOps → DevOps)
# Example: Model deployment specification
model_deployment:
name: customer-churn-model
version: 2.1.0
container:
image: mlplatform/models:customer-churn-v2.1.0
resources:
cpu: "2"
memory: "4Gi"
gpu: "0"
scaling:
min_replicas: 2
max_replicas: 10
target_cpu_utilization: 70%
endpoints:
health: /health
predict: /predict
metrics: /metrics
monitoring:
latency_p99_threshold_ms: 100
error_rate_threshold: 0.01
Key Takeaways
Summary
- DevOps focuses on application delivery - MLOps extends this to models
- DataOps focuses on data quality - MLOps depends on clean, versioned data
- MLOps combines both and adds ML-specific practices (experiment tracking, model versioning)
- Team structure should match organizational maturity and size
- Clear interfaces between teams prevent bottlenecks and confusion
Practical Exercise
Create a simple comparison document for your organization:
- List current DevOps practices
- List current DataOps practices (if any)
- Identify gaps for MLOps implementation
- Propose team structure changes