Skip to main content

Chapter 1: Transformers Library

The Foundation of Modern LLM Training​

The Transformers library provides everything needed to load, configure, and train language models.

Core Components​

AutoModelForCausalLM​

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

AutoTokenizer​

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

Model Loading Options​

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    # Precision
    torch_dtype=torch.bfloat16,
    
    # Device placement
    device_map="auto",  # or "cuda:0"
    
    # Quantization
    quantization_config=bnb_config,
    
    # Flash Attention 2
    attn_implementation="flash_attention_2",
    
    # Remote code
    trust_remote_code=True,
)

Tokenization​

# Basic tokenization
inputs = tokenizer("Hello, world!", return_tensors="pt")

# With padding and truncation
inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt",
)

# Chat template
messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False)

Key Takeaways​

✅ AutoModelForCausalLM for text generation models ✅ Use torch_dtype=bfloat16 for efficiency ✅ device_map="auto" for automatic placement ✅ Set pad_token = eos_token for training

Continue to Chapter 2 →