Chapter 3: Training Arguments and Optimization

Fine-Tuning Your Training Configuration

Optimal training arguments are crucial for successful fine-tuning.

---

Complete TrainingArguments

from transformers import TrainingArguments

training_args = TrainingArguments(
    # Output
    output_dir="./results",
    overwrite_output_dir=True,
    
    # Training duration
    num_train_epochs=3,
    max_steps=-1,  # -1 means use epochs
    
    # Batch size
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,
    
    # Learning rate
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    
    # Precision
    bf16=True,
    
    # Memory optimization
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",
    
    # Logging
    logging_steps=10,
    logging_first_step=True,
    
    # Saving
    save_strategy="epoch",
    save_total_limit=2,
    
    # Evaluation
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
    
    # Other
    seed=42,
    dataloader_num_workers=4,
)

---

Learning Rate Schedules

Schedule	Behavior	Best For
linear	Linear decay	Simple tasks
cosine	Cosine decay	Most fine-tuning
constant	No decay	Very short training
polynomial	Polynomial decay	Longer training

---

Hyperparameter Guidelines

Parameter	LoRA	QLoRA
Learning rate	1e-4 to 5e-4	1e-4 to 2e-4
Epochs	1-5	1-3
Batch size (effective)	8-32	8-16
Warmup	3-10%	3-5%

---

Troubleshooting

Issue	Solution
Loss not decreasing	Increase learning rate
NaN loss	Decrease LR, enable grad clipping
OOM error	Reduce batch size, enable checkpointing
Overfitting	Early stopping, more data, regularization

---

Continue to Quiz →