Chapter 1: Full Fine-Tuning

When to Use Full Fine-Tuning

• Large domain shift from pre-training data
• Sufficient compute and data available
• Maximum performance required

Compute Requirements

Model Size	VRAM (FP16)	VRAM (LoRA)
7B	56GB	16GB
13B	104GB	32GB
70B	560GB	80GB

Training Loop

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    warmup_steps=100,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

Catastrophic Forgetting

Full fine-tuning can cause the model to forget pre-trained knowledge. Mitigations:

• Lower learning rates
• Shorter training
• Regularization
• Replay buffers

---

← Module OverviewChapter 2 →