Skip to main content

Chapter 3: Data Quality and Validation

Great Expectations​

Great Expectations is a Python library for validating, documenting, and profiling data.

Setup and Basic Usage​

import great_expectations as gx
from great_expectations.core.batch import BatchRequest

# Create context
context = gx.get_context()

# Create expectation suite
suite = context.create_expectation_suite("customer_data_suite")

# Get validator
validator = context.get_validator(
    batch_request=BatchRequest(
        datasource_name="my_datasource",
        data_connector_name="default",
        data_asset_name="customers"
    ),
    expectation_suite_name="customer_data_suite"
)

# Add expectations
validator.expect_column_values_to_not_be_null("customer_id")
validator.expect_column_values_to_be_between("age", min_value=18, max_value=120)
validator.expect_column_values_to_be_unique("email")
validator.expect_column_values_to_match_regex("phone", r"^\+?[0-9]{10,15}$")

# Run validation
results = validator.validate()
print(f"Success: {results.success}")

Data Contracts​

# data_contract.yaml
contract:
  name: customer_features
  version: "1.0.0"
  owner: data-engineering@company.com
  
  schema:
    customer_id:
      type: string
      nullable: false
      unique: true
    age:
      type: integer
      min: 18
      max: 120
    email:
      type: string
      format: email
      nullable: false
      
  quality:
    completeness: 0.99
    freshness: "1 hour"
    
  sla:
    availability: "99.9%"
    update_frequency: "hourly"

Key Takeaways​

Summary
  1. Validate data at every stage of the pipeline
  2. Use data contracts between teams
  3. Monitor for drift continuously
  4. Fail fast on quality issues

Next: Module 4 Quiz