Capítulo 3: Comparando Experimentos

Comparação Eficaz de Experimentos

Comparar experimentos de forma eficaz é fundamental para selecionar o melhor modelo e entender o que impulsiona o desempenho.

Comparação de Experimentos no MLflow

import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient()

# Get all runs from an experiment
experiment = client.get_experiment_by_name("customer-churn")
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string="metrics.auc_roc > 0.8",
    order_by=["metrics.auc_roc DESC"],
    max_results=10
)

# Compare runs
comparison = []
for run in runs:
    comparison.append({
        "run_id": run.info.run_id,
        "run_name": run.data.tags.get("mlflow.runName", ""),
        "auc_roc": run.data.metrics.get("auc_roc"),
        "accuracy": run.data.metrics.get("accuracy"),
        "n_estimators": run.data.params.get("n_estimators"),
        "max_depth": run.data.params.get("max_depth")
    })

import pandas as pd
df_comparison = pd.DataFrame(comparison)
print(df_comparison.to_markdown())

Boas Práticas de Visualização

import matplotlib.pyplot as plt
import seaborn as sns

# Parallel coordinates plot for hyperparameter analysis
from pandas.plotting import parallel_coordinates

fig, ax = plt.subplots(figsize=(12, 6))
parallel_coordinates(
    df_comparison[['n_estimators', 'max_depth', 'auc_roc']], 
    'auc_roc',
    colormap='viridis'
)
plt.title("Hyperparameter Impact on AUC-ROC")
plt.savefig("hyperparameter_analysis.png")
mlflow.log_artifact("hyperparameter_analysis.png")

Framework de Seleção de Experimentos

class ExperimentSelector:
    """Framework for selecting the best experiment."""
    
    def __init__(self, experiments: list, weights: dict):
        self.experiments = experiments
        self.weights = weights  # e.g., {"auc_roc": 0.5, "latency": 0.3, "model_size": 0.2}
    
    def calculate_score(self, exp: dict) -> float:
        """Calculate weighted score for experiment."""
        score = 0
        for metric, weight in self.weights.items():
            value = exp.get(metric, 0)
            # Normalize (assume higher is better except for latency/size)
            if metric in ["latency", "model_size"]:
                value = 1 / (value + 1)  # Invert
            score += value * weight
        return score
    
    def select_best(self) -> dict:
        """Select best experiment based on weighted criteria."""
        scored = [(exp, self.calculate_score(exp)) for exp in self.experiments]
        return max(scored, key=lambda x: x[1])[0]

Principais Aprendizados

Summary

1. Use consultas estruturadas para filtrar e ordenar experimentos
2. Visualize o impacto dos hiperparâmetros com coordenadas paralelas
3. Defina critérios de seleção com métricas ponderadas
4. Documente as decisões para reprodutibilidade
5. Automatize a comparação em pipelines de CI/CD

---

Próximo: Quiz do Módulo 3