Capítulo 3: Implementación de LoRA con PEFT

Entrenamiento Práctico con LoRA

Este capítulo proporciona un código completo y listo para producción para implementar fine-tuning con LoRA utilizando la librería PEFT de Hugging Face.

---

Script de Entrenamiento Completo

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
)
from peft import (
    LoraConfig,
    get_peft_model,
    TaskType,
    prepare_model_for_kbit_training,
)
from trl import SFTTrainer
from datasets import load_dataset

# ============================================
# Configuration
# ============================================

MODEL_NAME = "meta-llama/Llama-3.1-8B"
OUTPUT_DIR = "./lora-output"
DATASET = "databricks/dolly-15k"

# LoRA hyperparameters
LORA_R = 16
LORA_ALPHA = 32
LORA_DROPOUT = 0.05
TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Training hyperparameters
LEARNING_RATE = 2e-4
BATCH_SIZE = 4
GRADIENT_ACCUMULATION = 4
MAX_SEQ_LENGTH = 512
NUM_EPOCHS = 3

# ============================================
# Load Model and Tokenizer
# ============================================

print("Loading model and tokenizer...")

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# ============================================
# Configure LoRA
# ============================================

print("Configuring LoRA...")

lora_config = LoraConfig(
    r=LORA_R,
    lora_alpha=LORA_ALPHA,
    lora_dropout=LORA_DROPOUT,
    target_modules=TARGET_MODULES,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Apply LoRA
model = get_peft_model(model, lora_config)

# Print trainable parameters
model.print_trainable_parameters()

# ============================================
# Prepare Dataset
# ============================================

print("Loading dataset...")

dataset = load_dataset(DATASET, split="train")

def format_instruction(example):
    """Format data for instruction tuning."""
    if example.get("context"):
        text = f"""### Instruction:
{example['instruction']}

### Context:
{example['context']}

### Response:
{example['response']}"""
    else:
        text = f"""### Instruction:
{example['instruction']}

### Response:
{example['response']}"""
    return {"text": text}

dataset = dataset.map(format_instruction)

# ============================================
# Training Arguments
# ============================================

training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=NUM_EPOCHS,
    per_device_train_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRADIENT_ACCUMULATION,
    learning_rate=LEARNING_RATE,
    weight_decay=0.01,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    gradient_checkpointing=True,
    optim="adamw_torch_fused",
    report_to="none",  # or "wandb"
    seed=42,
)

# ============================================
# Create Trainer
# ============================================

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    packing=False,
)

# ============================================
# Train
# ============================================

print("Starting training...")
trainer.train()

# ============================================
# Save
# ============================================

print("Saving model...")
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)

print(f"Training complete! Model saved to {OUTPUT_DIR}")

---

Parámetros de LoraConfig Explicados

Parámetros Principales

LoraConfig(
    # Rank - capacidad de adaptación
    r=16,  # Típico: 8, 16, 32, 64
    
    # Factor de escalado
    lora_alpha=32,  # Típico: 1x a 2x de r
    
    # Regularización
    lora_dropout=0.05,  # Típico: 0.0 a 0.1
    
    # Capas a adaptar
    target_modules=["q_proj", "v_proj"],
    
    # Entrenamiento de bias
    bias="none",  # "none", "all", o "lora_only"
    
    # Tipo de tarea
    task_type=TaskType.CAUSAL_LM,
)

Módulos Objetivo por Modelo

# Llama / Mistral
TARGET_MODULES_LLAMA = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]

# GPT-2 / GPT-J
TARGET_MODULES_GPT = [
    "c_attn", "c_proj", "c_fc",
]

# BERT / RoBERTa
TARGET_MODULES_BERT = [
    "query", "key", "value",
    "dense",
]

---

Carga y Uso de un Adaptador Entrenado

Carga Básica

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# Load adapter
model = PeftModel.from_pretrained(base_model, "./lora-output")

# Generate
inputs = tokenizer("Write a haiku about coding:", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Fusión del Adaptador

# Merge for deployment (no adapter overhead)
model = model.merge_and_unload()

# Save merged model
model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")

---

Entrenamiento Multi-Adaptador

Entrenamiento de Múltiples Adaptadores

from peft import LoraConfig, get_peft_model

# Create base PEFT model
model = get_peft_model(base_model, lora_config)

# Train first adapter
trainer.train()
model.save_pretrained("./adapter-task-a")

# Add another adapter for a different task
model.add_adapter("task_b", lora_config)
model.set_adapter("task_b")

# Train second adapter
trainer.train()  # With different dataset
model.save_pretrained("./adapter-task-b", save_adapter="task_b")

Cambio de Adaptadores

# Load base with first adapter
model = PeftModel.from_pretrained(base_model, "./adapter-task-a")

# Load additional adapter
model.load_adapter("./adapter-task-b", adapter_name="task_b")

# Switch between adapters
model.set_adapter("default")  # Use task A
output_a = model.generate(...)

model.set_adapter("task_b")  # Use task B
output_b = model.generate(...)

---

Guía de Ajuste de Hiperparámetros

Selección de Rank

# Start here
rank_experiments = [8, 16, 32]

for r in rank_experiments:
    config = LoraConfig(r=r, lora_alpha=r*2, ...)
    # Train and evaluate
    # Compare quality vs training time

Tasa de Aprendizaje

# LoRA typically uses higher LR than full FT
lr_options = [1e-4, 2e-4, 5e-4]

# Start with 2e-4, adjust based on:
# - Loss not decreasing → increase LR
# - Training unstable → decrease LR

Módulos Objetivo

# Minimal (fastest, lowest quality)
minimal = ["q_proj", "v_proj"]

# Standard (good balance)
standard = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Full (highest quality, most params)
full = ["q_proj", "k_proj", "v_proj", "o_proj", 
        "gate_proj", "up_proj", "down_proj"]

---

Depuración de Problemas Comunes

Problema: La Pérdida No Disminuye

# Check 1: Is adapter trainable?
model.print_trainable_parameters()
# Should show non-zero trainable params

# Check 2: Learning rate too low?
# Try increasing: 1e-4 → 2e-4 → 5e-4

# Check 3: Data formatting correct?
print(dataset[0]["text"])  # Verify format

Problema: Pérdida NaN

# Solution 1: Lower learning rate
learning_rate = 1e-5

# Solution 2: Use gradient clipping
training_args = TrainingArguments(
    max_grad_norm=1.0,
    ...
)

# Solution 3: Check for inf/nan in data

Problema: Falta de Memoria

# Solution 1: Reduce batch size
per_device_train_batch_size = 1

# Solution 2: Increase gradient accumulation
gradient_accumulation_steps = 16

# Solution 3: Enable gradient checkpointing
gradient_checkpointing = True

# Solution 4: Use QLoRA instead (next module)

---

Lista de Verificación para Producción

□ Elegir un rank apropiado (empezar con 16)
□ Establecer alpha en 2x rank
□ Apuntar al menos a todas las proyecciones de atención
□ Usar una tasa de aprendizaje adecuada (1e-4 a 5e-4)
□ Habilitar gradient checkpointing para memoria
□ Configurar métricas de evaluación
□ Guardar checkpoints regularmente
□ Fusionar el adaptador para despliegue
□ Probar el modelo fusionado exhaustivamente

---

Puntos Clave

✅ La librería PEFT facilita la implementación de LoRA

✅ Configuración clave: r (rank), alpha (escalado), target_modules

✅ SFTTrainer simplifica el ciclo de entrenamiento

✅ Los adaptadores pueden guardarse/cargarse/fusionarse de forma independiente

✅ Múltiples adaptadores permiten el cambio de tareas

✅ Comience con configuraciones estándar y ajuste según sea necesario

---

Próximos Pasos

Ponga a prueba sus conocimientos con el Cuestionario del Módulo 4, luego continúe con Módulo 5: QLoRA y Cuantización.

---

Verificación Rápida

Antes de continuar, asegúrese de poder responder:

1. ¿Qué función crea un modelo PEFT a partir de un modelo base?
2. ¿Cómo se fusiona un adaptador LoRA en el modelo base?
3. ¿Cuál es una tasa de aprendizaje típica para el entrenamiento con LoRA?