Chapter 2: Data Version Control (DVC)

Introduction

DVC (Data Version Control) extends Git's capabilities to handle large files, datasets, and machine learning models. It enables reproducibility by tracking data and models alongside code.

Why DVC?

The Problem with Large Files in Git

DVC Benefits

Feature	Benefit
Git-like commands	Familiar workflow for developers
External storage	Data stays outside Git
Deduplication	Same data stored once
Pipelines	Reproducible ML workflows
Remote storage	S3, GCS, Azure, SSH support

Setting Up DVC

Installation

# Using pip
pip install dvc

# With remote storage support
pip install "dvc[s3]"     # Amazon S3
pip install "dvc[gs]"     # Google Cloud Storage
pip install "dvc[azure]"  # Azure Blob Storage
pip install "dvc[ssh]"    # SSH remote
pip install "dvc[all]"    # All remotes

# Initialize DVC in your project
cd your-project
dvc init
git add .dvc .dvcignore
git commit -m "Initialize DVC"

DVC Directory Structure

After initialization:

your-project/
├── .dvc/
│   ├── config        # DVC configuration
│   ├── .gitignore    # Files DVC ignores in its own folder
│   └── cache/        # Local cache (gitignored)
├── .dvcignore        # Files DVC should ignore
└── ...

Tracking Data with DVC

Basic Data Tracking

# Add a data file to DVC tracking
dvc add data/training_data.csv

# This creates:
# - data/training_data.csv.dvc (metadata file)
# - data/.gitignore (prevents git from tracking the actual file)

# Add the DVC files to git
git add data/training_data.csv.dvc data/.gitignore
git commit -m "data: track training data with DVC"

Understanding .dvc Files

# data/training_data.csv.dvc
outs:
- md5: a3f2b1c4d5e6f7g8h9i0j1k2
  size: 1073741824
  path: training_data.csv

The .dvc file contains:

• md5: Hash of the file content
• size: File size in bytes
• path: Relative path to the file

Tracking Directories

# Track entire directories
dvc add data/images/

# Creates data/images.dvc
git add data/images.dvc data/.gitignore
git commit -m "data: add image dataset"

Configuring Remote Storage

Setting Up S3 Remote

# Add S3 remote
dvc remote add -d myremote s3://mybucket/dvc-storage

# Configure credentials (or use IAM roles)
dvc remote modify myremote access_key_id 'YOUR_ACCESS_KEY'
dvc remote modify myremote secret_access_key 'YOUR_SECRET_KEY'

# Or use AWS profile
dvc remote modify myremote profile myprofile

# Enable cache for faster operations
dvc remote modify myremote cache true

Setting Up GCS Remote

# Add Google Cloud Storage remote
dvc remote add -d myremote gs://mybucket/dvc-storage

# Authenticate using service account
dvc remote modify myremote credentialpath /path/to/credentials.json

Setting Up Azure Blob Remote

# Add Azure Blob Storage remote
dvc remote add -d myremote azure://mycontainer/dvc-storage

# Configure account
dvc remote modify myremote account_name 'myaccount'
dvc remote modify myremote account_key 'mykey'

Local Remote (for development)

# Use local directory as remote (useful for development)
dvc remote add -d local /shared/dvc-storage

# Or network drive
dvc remote add -d network //server/dvc-storage

Pushing and Pulling Data

Push Data to Remote

# Push all tracked data to default remote
dvc push

# Push specific file
dvc push data/training_data.csv.dvc

# Push to specific remote
dvc push -r myremote

Pull Data from Remote

# Pull all data from default remote
dvc pull

# Pull specific file
dvc pull data/training_data.csv.dvc

# Pull from specific remote
dvc pull -r myremote

Checkout Specific Version

# Switch to different git commit/branch
git checkout v1.0.0

# Checkout corresponding data version
dvc checkout

DVC Pipelines

Why Pipelines?

Pipelines define the sequence of data processing steps, making experiments reproducible.

Defining Pipelines

# dvc.yaml
stages:
  prepare:
    cmd: python src/data/prepare.py
    deps:
      - src/data/prepare.py
      - data/raw/
    params:
      - prepare.seed
      - prepare.split_ratio
    outs:
      - data/processed/train.csv
      - data/processed/test.csv

  featurize:
    cmd: python src/features/featurize.py
    deps:
      - src/features/featurize.py
      - data/processed/train.csv
      - data/processed/test.csv
    params:
      - featurize.max_features
    outs:
      - data/features/train_features.pkl
      - data/features/test_features.pkl

  train:
    cmd: python src/models/train.py
    deps:
      - src/models/train.py
      - data/features/train_features.pkl
    params:
      - train.n_estimators
      - train.max_depth
      - train.learning_rate
    outs:
      - models/model.pkl
    metrics:
      - metrics/train_metrics.json:
          cache: false

  evaluate:
    cmd: python src/models/evaluate.py
    deps:
      - src/models/evaluate.py
      - models/model.pkl
      - data/features/test_features.pkl
    metrics:
      - metrics/eval_metrics.json:
          cache: false
    plots:
      - metrics/plots/:
          cache: false

Parameters File

# params.yaml
prepare:
  seed: 42
  split_ratio: 0.2

featurize:
  max_features: 1000

train:
  n_estimators: 100
  max_depth: 10
  learning_rate: 0.1

Running Pipelines

# Run entire pipeline
dvc repro

# Run specific stage
dvc repro train

# Force run (ignore cache)
dvc repro -f

# Visualize pipeline
dvc dag

Pipeline Visualization

$ dvc dag
         +----------+
         | prepare  |
         +----------+
              *
              *
              *
        +-----------+
        | featurize |
        +-----------+
              *
              *
              *
         +---------+
         |  train  |
         +---------+
              *
              *
              *
        +----------+
        | evaluate |
        +----------+

Experiments with DVC

Running Experiments

# Run experiment with modified parameters
dvc exp run --set-param train.n_estimators=200

# Run multiple experiments in queue
dvc exp run --queue --set-param train.n_estimators=100
dvc exp run --queue --set-param train.n_estimators=200
dvc exp run --queue --set-param train.n_estimators=300

# Run queued experiments
dvc exp run --run-all

# List experiments
dvc exp show

Comparing Experiments

# Show experiments with metrics
dvc exp show --md

# Compare specific experiments
dvc exp diff exp-abc123 exp-def456

Managing Experiments

# Apply experiment to workspace
dvc exp apply exp-abc123

# Create branch from experiment
dvc exp branch exp-abc123 feature/best-model

# Remove experiments
dvc exp remove exp-abc123

DVC Metrics and Plots

Defining Metrics

# src/models/train.py
import json

metrics = {
    "accuracy": 0.92,
    "f1_score": 0.89,
    "auc_roc": 0.95
}

with open("metrics/train_metrics.json", "w") as f:
    json.dump(metrics, f)

Viewing Metrics

# Show metrics
dvc metrics show

# Compare metrics across commits
dvc metrics diff HEAD~1

Creating Plots

# src/models/evaluate.py
import json

# ROC curve data
roc_data = [
    {"fpr": 0.0, "tpr": 0.0},
    {"fpr": 0.1, "tpr": 0.6},
    {"fpr": 0.2, "tpr": 0.8},
    {"fpr": 1.0, "tpr": 1.0}
]

with open("metrics/plots/roc.json", "w") as f:
    json.dump(roc_data, f)

# Generate plots
dvc plots show

# Compare plots across experiments
dvc plots diff exp-abc123 exp-def456

Best Practices

1. Organize Data Directories

data/
├── raw/           # Original, immutable data
├── interim/       # Intermediate transformations
├── processed/     # Final, clean data for modeling
└── external/      # Data from external sources

2. Use Meaningful .dvc File Names

# Good: Descriptive names
dvc add data/customer_transactions_2024.parquet

# Avoid: Generic names
dvc add data/data.csv

3. Version Data with Metadata

# Tag data versions with git
git tag -a "data-v1.0.0" -m "Initial training dataset"
git push origin data-v1.0.0

4. Document Data Sources

# data/README.md or data/sources.yaml
sources:
  customer_transactions:
    description: "Customer transaction history"
    source: "PostgreSQL production database"
    refresh_frequency: "daily"
    columns:
      - name: transaction_id
        type: string
        description: "Unique transaction identifier"
      - name: amount
        type: float
        description: "Transaction amount in USD"

5. Automate DVC Operations

# .github/workflows/dvc.yml
name: DVC Pipeline
on:
  push:
    branches: [main]

jobs:
  dvc-pipeline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: iterative/setup-dvc@v1
      
      - name: Configure DVC remote
        run: |
          dvc remote modify --local myremote access_key_id ${{ secrets.AWS_ACCESS_KEY }}
          dvc remote modify --local myremote secret_access_key ${{ secrets.AWS_SECRET_KEY }}
      
      - name: Pull data
        run: dvc pull
      
      - name: Run pipeline
        run: dvc repro
      
      - name: Push results
        run: dvc push

Troubleshooting

Common Issues

# Error: "Unable to acquire lock"
rm .dvc/lock

# Error: "Checkout failed"
dvc checkout -f

# Error: "Push failed" (large files)
dvc push --jobs 4  # Parallel uploads

# Check cache status
dvc cache dir

# Clean unused cache
dvc gc -w

Key Takeaways

Summary

1. DVC extends Git for large files without bloating the repository
2. Use remote storage for team collaboration (S3, GCS, Azure)
3. Define pipelines for reproducible ML workflows
4. Track experiments with DVC exp commands
5. Version data alongside code for full reproducibility
6. Automate DVC operations in CI/CD pipelines

Practical Exercise

1. Initialize DVC in a new project
2. Configure S3 or local remote storage
3. Track a dataset with DVC
4. Create a simple 3-stage pipeline
5. Run experiments with different parameters

---

Next: Chapter 3: Model Versioning