Chapitre 2 : Pipelines d'évaluation humaine

Critères d'évaluation

Critère	Description	Échelle
Utilité	Répond-il à la question ?	1-5
Exactitude	L'information est-elle correcte ?	1-5
Innocuité	Est-ce sûr et approprié ?	Oui/Non
Cohérence	Est-ce bien structuré ?	1-5

Interface d'annotation

# Example annotation task structure
annotation_task = {
    "task_id": "task_001",
    "prompt": "What are the benefits of MLOps?",
    "response_a": "MLOps provides...",
    "response_b": "The benefits include...",
    "evaluation_criteria": [
        {"name": "helpfulness", "type": "rating", "scale": [1, 2, 3, 4, 5]},
        {"name": "accuracy", "type": "rating", "scale": [1, 2, 3, 4, 5]},
        {"name": "preference", "type": "choice", "options": ["A", "B", "Tie"]}
    ]
}

Accord inter-annotateurs

from sklearn.metrics import cohen_kappa_score

def calculate_agreement(annotator_1: list, annotator_2: list) -> float:
    """Calculate Cohen's Kappa for inter-annotator agreement."""
    kappa = cohen_kappa_score(annotator_1, annotator_2)
    
    # Interpretation
    if kappa < 0.20:
        interpretation = "Poor"
    elif kappa < 0.40:
        interpretation = "Fair"
    elif kappa < 0.60:
        interpretation = "Moderate"
    elif kappa < 0.80:
        interpretation = "Good"
    else:
        interpretation = "Excellent"
    
    return kappa, interpretation

---

Suivant : Chapitre 3 : Tests de sécurité