Skip to main content

Chapter 3: Memory-Efficient Training

Optimizing Training for Limited Hardware​

This chapter covers all techniques for maximizing training efficiency on limited GPU memory.

Memory Optimization Techniques​

1. Gradient Checkpointing​

Trade compute for memory by recomputing activations:

# Enable via training args
training_args = TrainingArguments(
    gradient_checkpointing=True,
    ...
)

# Or directly on model
model.gradient_checkpointing_enable()

Savings: ~30% memory reduction Cost: ~20% slower training

2. Gradient Accumulation​

Simulate larger batches with limited memory:

training_args = TrainingArguments(
    per_device_train_batch_size=1,      # Small batch fits in memory
    gradient_accumulation_steps=16,     # Accumulate gradients
    # Effective batch size = 1 * 16 = 16
)

3. Efficient Optimizers​

8-bit Adam​

training_args = TrainingArguments(
    optim="adamw_8bit",  # Requires bitsandbytes
)

Paged Optimizers​

training_args = TrainingArguments(
    optim="paged_adamw_8bit",  # Pages to CPU when needed
)

Complete Memory-Efficient Training Script​

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset

# Quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer.pad_token = tokenizer.eos_token

# Prepare for training
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

# Dataset
dataset = load_dataset("databricks/dolly-15k", split="train[:1000]")

# Memory-efficient training args
training_args = TrainingArguments(
    output_dir="./qlora-output",
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    bf16=True,
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",
    logging_steps=10,
    save_strategy="epoch",
)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
)

trainer.train()
model.save_pretrained("./qlora-adapter")

Memory Checklist​

□ Use QLoRA (4-bit quantization)
□ Enable gradient checkpointing
□ Use gradient accumulation
□ Use 8-bit or paged optimizer
□ Use bfloat16 precision
□ Reduce max_seq_length if possible
□ Use smaller batch sizes

Expected Memory Usage​

ModelWithout OptimizationFull Optimization
7B~14GB~4-6GB
13B~26GB~8-10GB
70B~140GB~40-48GB

Key Takeaways​

✅ Combine multiple optimization techniques ✅ Gradient checkpointing trades compute for memory ✅ Gradient accumulation simulates larger batches ✅ Paged optimizers handle memory spikes ✅ QLoRA + all optimizations enable 7B on 6GB

Continue to Quiz →