Chapter 1: Pre-training Objectives

Introduction

Pre-training is the process of training a language model on large amounts of unlabeled text data. The key insight is that predicting text itself provides a rich learning signal.

Next Token Prediction (Causal LM)

The Core Idea

Given a sequence of tokens, predict the next token:

$$P(x_t | x_1, x_2, ..., x_{t-1})$$

Implementation

import torch
import torch.nn.functional as F

def causal_lm_loss(logits, targets):
    """
    Compute causal language modeling loss.
    logits: (batch, seq_len, vocab_size)
    targets: (batch, seq_len)
    """
    # Shift: predict next token from each position
    shift_logits = logits[..., :-1, :].contiguous()
    shift_targets = targets[..., 1:].contiguous()
    
    # Cross-entropy loss
    loss = F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_targets.view(-1),
        ignore_index=-100  # Padding token
    )
    return loss

Used By

• GPT series (OpenAI)
• LLaMA (Meta)
• Mistral

Masked Language Modeling (MLM)

The BERT Approach

Randomly mask tokens and predict them:

def create_mlm_data(input_ids, tokenizer, mask_prob=0.15):
    """Create masked language modeling training data."""
    labels = input_ids.clone()
    
    # Create random mask
    probability_matrix = torch.full(labels.shape, mask_prob)
    masked_indices = torch.bernoulli(probability_matrix).bool()
    
    # Don't mask special tokens
    special_tokens_mask = [
        tokenizer.get_special_tokens_mask(val) 
        for val in labels.tolist()
    ]
    masked_indices &= ~torch.tensor(special_tokens_mask).bool()
    
    # Replace with [MASK] token
    input_ids[masked_indices] = tokenizer.mask_token_id
    
    # Only compute loss on masked tokens
    labels[~masked_indices] = -100
    
    return input_ids, labels

Used By

• BERT
• RoBERTa
• Modern encoder models

Causal vs Bidirectional

Aspect	Causal (GPT)	Bidirectional (BERT)
Context	Left only	Both directions
Best for	Generation	Understanding
Training	Autoregressive	Masked

Key Takeaways

Summary

1. Next token prediction enables autoregressive generation
2. Masked LM provides bidirectional context understanding
3. Causal models dominate modern LLMs for their generation capability

---

← Module Overview

Chapter 2: Data Collection →