Chapter 1: Git for ML Projects

Introduction

Git is the foundation of version control in software development, but ML projects introduce unique challenges that require specialized practices and configurations.

ML-Specific Git Challenges

The Unique Nature of ML Code

Unlike traditional software, ML projects involve:

Challenge	Description
Large files	Datasets, model weights (GB to TB)
Binary artifacts	Trained models, preprocessors
Experiment sprawl	Hundreds of experimental branches
Non-determinism	Same code may produce different results
Notebook chaos	Jupyter notebooks with embedded outputs

Repository Structure for ML Projects

Recommended Structure

ml-project/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── training.yml
│       └── deployment.yml
├── data/
│   ├── raw/              # Original data (DVC tracked)
│   ├── processed/        # Transformed data (DVC tracked)
│   └── features/         # Feature store exports
├── models/
│   ├── artifacts/        # Trained models (DVC tracked)
│   └── configs/          # Model configurations
├── notebooks/
│   ├── exploration/      # EDA notebooks
│   └── experiments/      # Experiment notebooks
├── src/
│   ├── __init__.py
│   ├── data/
│   │   ├── __init__.py
│   │   ├── ingestion.py
│   │   └── preprocessing.py
│   ├── features/
│   │   ├── __init__.py
│   │   └── engineering.py
│   ├── models/
│   │   ├── __init__.py
│   │   ├── train.py
│   │   └── predict.py
│   └── utils/
│       ├── __init__.py
│       └── helpers.py
├── tests/
│   ├── unit/
│   ├── integration/
│   └── data/
├── configs/
│   ├── training_config.yaml
│   └── serving_config.yaml
├── .dvc/
├── .gitignore
├── .pre-commit-config.yaml
├── dvc.yaml
├── dvc.lock
├── pyproject.toml
├── requirements.txt
└── README.md

Comprehensive .gitignore for ML

# .gitignore for ML Projects

# ===== Python =====
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# ===== Virtual Environments =====
venv/
ENV/
env/
.venv/
.conda/

# ===== Jupyter Notebooks =====
.ipynb_checkpoints/
*/.ipynb_checkpoints/

# ===== Data Files =====
# Large data files (tracked by DVC instead)
*.csv
*.parquet
*.feather
*.h5
*.hdf5
*.pkl
*.pickle
!data/.gitkeep
data/raw/*
data/processed/*
data/features/*

# ===== Model Artifacts =====
# Trained models (tracked by DVC instead)
*.pt
*.pth
*.onnx
*.pb
*.h5
*.keras
*.joblib
*.model
models/artifacts/*
!models/artifacts/.gitkeep

# ===== MLOps Artifacts =====
mlruns/
outputs/
wandb/
.neptune/
tensorboard_logs/

# ===== IDE =====
.idea/
.vscode/
*.swp
*.swo
.DS_Store

# ===== Environment Variables =====
.env
.env.local
*.env

# ===== Secrets =====
secrets/
credentials/
*.pem
*.key

# ===== Logs =====
logs/
*.log

# ===== Cache =====
.cache/
.pytest_cache/
.mypy_cache/
.ruff_cache/

# ===== Coverage =====
htmlcov/
.coverage
.coverage.*
coverage.xml

Git LFS for Large Files

When to Use Git LFS

Git LFS (Large File Storage) is useful for files that:

• Are essential to the repository
• Are updated infrequently
• Are smaller than 2GB
• Cannot be tracked by DVC

Setting Up Git LFS

# Install Git LFS
git lfs install

# Track specific file types
git lfs track "*.bin"
git lfs track "*.weights"
git lfs track "*.ckpt"

# Track specific directories
git lfs track "models/pretrained/*"

# Verify tracking
cat .gitattributes

.gitattributes Example

# Git LFS tracking
*.bin filter=lfs diff=lfs merge=lfs -text
*.weights filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
models/pretrained/* filter=lfs diff=lfs merge=lfs -text

# Ensure consistent line endings
*.py text eol=lf
*.yaml text eol=lf
*.yml text eol=lf
*.json text eol=lf
*.md text eol=lf

# Notebooks as binary (prevent merge conflicts)
*.ipynb binary

Branching Strategies for ML

GitFlow Adapted for ML

Branch Naming Conventions

# Feature branches (infrastructure, pipelines)
feature/data-pipeline
feature/monitoring-setup
feature/api-endpoint

# Experiment branches (model development)
experiment/baseline-rf
experiment/deep-learning-v1
experiment/transformer-tuning

# Hotfix branches
hotfix/data-drift-fix
hotfix/prediction-bug

# Release branches
release/v1.0.0
release/v1.1.0

ML-Specific Branch Rules

# .github/branch-protection.yml
branches:
  main:
    protection:
      required_reviews: 2
      required_status_checks:
        - unit-tests
        - integration-tests
        - model-validation
      enforce_admins: true
      
  develop:
    protection:
      required_reviews: 1
      required_status_checks:
        - unit-tests
        - linting
        
  experiment/*:
    protection:
      required_reviews: 0  # Experiments can be merged freely
      allow_force_push: true  # Allow rebasing experiments

Commit Message Conventions

Structured Commit Messages

# Format: <type>(<scope>): <description>

# Types for ML projects:
# - data: Data-related changes
# - feat: New feature
# - fix: Bug fix
# - model: Model architecture changes
# - train: Training-related changes
# - eval: Evaluation/metrics changes
# - docs: Documentation
# - refactor: Code refactoring
# - test: Tests
# - ci: CI/CD changes
# - config: Configuration changes

# Examples:
git commit -m "data: add new customer segmentation dataset v2.1"
git commit -m "model: implement attention mechanism in encoder"
git commit -m "train: add learning rate scheduling"
git commit -m "eval: add precision-recall analysis"
git commit -m "config: update training hyperparameters"

Commit Message Template

# .gitmessage
# <type>(<scope>): <subject>
# |<----  Using a Maximum Of 50 Characters  ---->|

# Explain why this change is being made
# |<----   Try To Limit Each Line to a Maximum Of 72 Characters   ---->|

# Provide links or keys to any relevant tickets, articles or other resources
# Example: Experiment: exp-2024-01-15-baseline
#          MLflow Run: abc123

# --- COMMIT END ---
# Type can be:
#   data     (data-related changes)
#   feat     (new feature)
#   fix      (bug fix)
#   model    (model architecture)
#   train    (training changes)
#   eval     (evaluation/metrics)
#   docs     (documentation)
#   refactor (refactoring code)
#   test     (adding tests)
#   ci       (CI/CD changes)
#   config   (configuration changes)

Set it as default:

git config --global commit.template ~/.gitmessage

Pre-commit Hooks for ML

Pre-commit Configuration

# .pre-commit-config.yaml
repos:
  # General Python
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-json
      - id: check-added-large-files
        args: ['--maxkb=1000']  # Prevent large files
      - id: detect-private-key
      - id: check-merge-conflict

  # Python formatting
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.1.9
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  # Type checking
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy
        additional_dependencies: [types-all]

  # Security
  - repo: https://github.com/PyCQA/bandit
    rev: 1.7.6
    hooks:
      - id: bandit
        args: ['-c', 'pyproject.toml']

  # Jupyter notebooks
  - repo: https://github.com/kynan/nbstripout
    rev: 0.6.1
    hooks:
      - id: nbstripout  # Strip notebook outputs before commit

  # DVC
  - repo: https://github.com/iterative/dvc
    rev: 3.38.1
    hooks:
      - id: dvc-pre-commit
      - id: dvc-pre-push

Installing Pre-commit

# Install pre-commit
pip install pre-commit

# Install hooks
pre-commit install

# Run on all files
pre-commit run --all-files

Handling Jupyter Notebooks

The Notebook Problem

Jupyter notebooks are problematic for version control:

• Contain output cells (images, dataframes)
• Include execution counts
• Large diffs due to metadata changes

Solution 1: Strip Outputs

# Using nbstripout
pip install nbstripout

# Install as git filter
nbstripout --install

# Strip manually
nbstripout notebook.ipynb

Solution 2: Jupytext for Notebooks

# Install jupytext
pip install jupytext

# Convert notebook to Python script
jupytext --to py:percent notebooks/experiment.ipynb

# Sync notebook and script
jupytext --sync notebooks/experiment.ipynb

# pyproject.toml - Jupytext configuration
[tool.jupytext]
formats = "notebooks///ipynb,scripts///py:percent"

Solution 3: ReviewNB for PR Reviews

Use ReviewNB for GitHub PR reviews of notebooks:

• Visual diffs for notebooks
• Comment on specific cells
• Track changes effectively

CI/CD Integration

GitHub Actions for ML

# .github/workflows/ml-ci.yml
name: ML CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for DVC

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          cache: 'pip'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r requirements-dev.txt

      - name: Run linting
        run: ruff check .

      - name: Run type checking
        run: mypy src/

      - name: Run unit tests
        run: pytest tests/unit/ -v --cov=src/

      - name: Upload coverage
        uses: codecov/codecov-action@v3

  data-validation:
    runs-on: ubuntu-latest
    needs: lint-and-test
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup DVC
        uses: iterative/setup-dvc@v1
        
      - name: Pull data
        run: dvc pull data/processed/
        
      - name: Validate data
        run: python src/data/validate.py

Key Takeaways

Summary

1. Structure your repository with clear separation of code, data, and models
2. Use comprehensive .gitignore to exclude large files and sensitive data
3. Consider Git LFS for essential large files that can't use DVC
4. Adopt ML-specific branching with experiment branches
5. Enforce commit conventions for better history tracking
6. Use pre-commit hooks to maintain code quality
7. Handle notebooks carefully with nbstripout or jupytext

Practical Exercise

1. Create a new ML repository with the recommended structure
2. Configure .gitignore and .gitattributes
3. Set up pre-commit hooks
4. Create and merge an experiment branch

---

Next: Chapter 2: Data Version Control (DVC)