Chapter 2: TRL and SFTTrainer

Simplified Training for LLMs

TRL (Transformer Reinforcement Learning) provides specialized trainers for fine-tuning.

---

SFTTrainer Basics

from trl import SFTTrainer
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    save_strategy="epoch",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",  # or use formatting_func
    max_seq_length=512,
    packing=False,
)

trainer.train()

---

Dataset Formatting

Option 1: Text Field

# Dataset has "text" column
trainer = SFTTrainer(
    dataset_text_field="text",
    ...
)

Option 2: Formatting Function

def formatting_func(example):
    return f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['response']}"

trainer = SFTTrainer(
    formatting_func=formatting_func,
    ...
)

---

Important Parameters

Parameter	Purpose	Typical Value
max_seq_length	Maximum sequence length	512-2048
packing	Pack multiple examples	False (simpler)
dataset_num_proc	Preprocessing workers	4-8

---

Callbacks and Logging

from transformers import EarlyStoppingCallback

trainer = SFTTrainer(
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
    ...
)

# For W&B logging
training_args = TrainingArguments(
    report_to="wandb",
    ...
)

---

Continue to Chapter 3 →