Chapter 1: Data Ingestion Pipelines

Introduction

Data ingestion is the foundation of any ML system. This chapter covers building reliable, scalable data pipelines using modern orchestration tools.

Apache Airflow for ML Pipelines

Basic DAG Structure

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'ml-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email': ['ml-alerts@company.com'],
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'ml_data_pipeline',
    default_args=default_args,
    description='Daily ML data ingestion pipeline',
    schedule_interval='0 6 * * *',  # Daily at 6 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['ml', 'data', 'production'],
) as dag:
    
    extract = PythonOperator(
        task_id='extract_data',
        python_callable=extract_from_source,
    )
    
    transform = PythonOperator(
        task_id='transform_data',
        python_callable=transform_data,
    )
    
    validate = PythonOperator(
        task_id='validate_data',
        python_callable=run_data_validation,
    )
    
    load = PythonOperator(
        task_id='load_to_feature_store',
        python_callable=load_features,
    )
    
    extract >> transform >> validate >> load

Streaming vs Batch Processing

Aspect	Batch Processing	Stream Processing
Latency	Minutes to hours	Milliseconds to seconds
Use Case	Training data, reports	Real-time features
Tools	Spark, Airflow	Kafka, Flink
Complexity	Lower	Higher

Key Takeaways

Summary

1. Use orchestration tools like Airflow for complex pipelines
2. Choose batch vs streaming based on latency requirements
3. Implement idempotency for reliable re-runs
4. Monitor pipeline health with alerts and dashboards

---

Next: Chapter 2: Feature Engineering at Scale