Glossary of LLM Terms
Over 200 technical terms used throughout this course.
A
AGI (Artificial General Intelligence): Hypothetical AI with human-level general cognitive abilities across all domains.
Attention: Mechanism allowing models to focus on relevant parts of input when producing output.
Autoregressive: Generation method predicting one token at a time, each conditioned on previous tokens.
AWQ (Activation-aware Weight Quantization): Quantization method that preserves important weights based on activation patterns.
Alignment: Training AI systems to behave according to human intentions and values.
Adapter: Small trainable modules inserted between frozen layers for parameter-efficient fine-tuning.
B
BPE (Byte Pair Encoding): Tokenization algorithm that iteratively merges frequent character pairs.
BERT: Bidirectional Encoder Representations from Transformers; encoder-only model using masked language modeling.
Batch Size: Number of samples processed together during training.
Beam Search: Decoding strategy maintaining multiple candidate sequences.
bitsandbytes: Library for on-the-fly quantization during inference.
C
Causal Attention: Attention where positions can only attend to previous positions (not future).
Chain-of-Thought (CoT): Prompting technique encouraging step-by-step reasoning.
Chinchilla: DeepMind model/paper showing compute-optimal training uses more data with smaller models.
Constitutional AI: Anthropic's approach training AI with principles rather than case-by-case labels.
Context Window: Maximum number of tokens a model can process at once.
Cross-Attention: Attention where queries come from one sequence and keys/values from another.
CLIP: Contrastive Language-Image Pre-training; model aligning images and text.
D
Data Parallelism: Training strategy where each GPU processes different data with same model.
Decoder-Only: Architecture (like GPT) using only decoder blocks with causal attention.
DeepSpeed: Microsoft library for distributed training with ZeRO optimization.
Distillation: Training smaller model to mimic larger model's behavior.
DPO (Direct Preference Optimization): Alternative to RLHF that directly optimizes on preference data.
E
Embedding: Dense vector representation of a token or sequence.
Emergent Capability: Ability appearing only at certain model scale, absent in smaller models.
Encoder-Decoder: Architecture with separate encoder (bidirectional) and decoder (causal).
Encoder-Only: Architecture (like BERT) using only encoder blocks with bidirectional attention.
F
Feed-Forward Network (FFN): MLP layers in Transformer blocks processing each position independently.
Few-Shot Learning: Learning from few examples provided in the prompt.
Fine-Tuning: Further training a pre-trained model on specific data/task.
Flash Attention: Memory-efficient attention implementation optimizing GPU memory access.
FSDP (Fully Sharded Data Parallel): PyTorch's implementation of ZeRO.
Function Calling: LLM capability to request external tool/API execution.
G
GPTQ: Post-training quantization method optimized for GPT-style models.
Gradient Accumulation: Simulating larger batch sizes by accumulating gradients over multiple steps.
Gradient Checkpointing: Trading compute for memory by recomputing activations during backward pass.
Greedy Decoding: Selecting highest probability token at each step.
H
Hallucination: Model generating false or unsupported information.
Hidden State: Internal representation at each layer of the model.
HumanEval: Benchmark for evaluating code generation ability.
I
In-Context Learning (ICL): Learning from examples in the prompt without weight updates.
Inference: Running a trained model to generate predictions.
Instruction Tuning: Fine-tuning on instruction-following data.
INT4/INT8: 4-bit and 8-bit integer quantization formats.
J
Jailbreaking: Attempting to bypass model safety measures through clever prompts.
K
KV Cache: Storing computed key-value pairs to avoid recomputation during generation.
KL Divergence: Measure of difference between probability distributions; used in RLHF.
L
Layer Normalization: Normalizing activations across features within each sample.
LLM (Large Language Model): Neural network with billions of parameters trained on vast text.
LoRA (Low-Rank Adaptation): PEFT method adding low-rank matrices to existing weights.
Loss Function: Measure of model's prediction error; minimized during training.
M
Masked Language Modeling (MLM): Pre-training by predicting masked tokens (used by BERT).
Mixed Precision: Using lower precision (FP16/BF16) for most computations.
MoE (Mixture of Experts): Architecture with multiple expert networks and a router.
Multi-Head Attention: Running multiple attention operations in parallel.
Multimodal: Models processing multiple modalities (text, images, audio).
N
Next Token Prediction: Pre-training objective predicting the next token given context.
Nucleus Sampling (Top-p): Sampling from tokens within top probability mass.
O
ONNX: Open Neural Network Exchange; framework-agnostic model format.
P
Parameter: Trainable weight in a neural network.
PEFT (Parameter-Efficient Fine-Tuning): Methods training small subset of parameters.
Perplexity: Measure of model's uncertainty; exponential of average cross-entropy loss.
Pipeline Parallelism: Placing different layers on different GPUs.
Positional Encoding: Information added to embeddings about token positions.
PPO (Proximal Policy Optimization): RL algorithm commonly used in RLHF.
Pre-training: Initial training on large unlabeled data.
Prompt: Input text provided to a language model.
Q
QLoRA: LoRA applied to quantized base model.
Quantization: Reducing numerical precision of weights to save memory.
Query, Key, Value (Q, K, V): Three projections used in attention computation.
R
RAG (Retrieval-Augmented Generation): Augmenting LLM with retrieved relevant context.
ReAct: Reasoning + Acting pattern for LLM agents.
Red Teaming: Adversarial testing to find model vulnerabilities.
Reward Model: Model trained to predict human preferences (used in RLHF).
RLHF (Reinforcement Learning from Human Feedback): Training with human preference data.
RoPE (Rotary Position Embedding): Position encoding through rotation of embeddings.
S
Scaling Laws: Predictable relationships between model size, data, and performance.
Self-Attention: Attention where a sequence attends to itself.
SentencePiece: Language-agnostic tokenization treating text as raw bytes.
SFT (Supervised Fine-Tuning): Fine-tuning on demonstration data.
Softmax: Function converting logits to probabilities summing to 1.
Speculative Decoding: Using small model to draft, large model to verify.
SSM (State Space Model): Architecture with linear complexity (e.g., Mamba).
Streaming: Returning partial results as they're generated.
T
Temperature: Parameter controlling output randomness (higher = more random).
Tensor Parallelism: Splitting weight matrices across GPUs.
TGI (Text Generation Inference): Hugging Face's production serving solution.
Token: Basic unit of text processed by language models.
Tokenization: Converting text to tokens (and vice versa).
Top-k Sampling: Sampling from top k highest probability tokens.
Transformer: Architecture based on self-attention, enabling parallel processing.
U
Unsupervised Learning: Learning from data without explicit labels.
V
vLLM: High-throughput serving framework with PagedAttention.
Vision Transformer (ViT): Transformer architecture for images using patches.
Vocabulary: Set of all tokens a model can process.
W
Weight: Trainable parameter in a neural network.
WordPiece: Tokenization algorithm used by BERT.
Z
Zero-Shot: Performing tasks without any examples in the prompt.
ZeRO (Zero Redundancy Optimizer): DeepSpeed optimization partitioning optimizer states/gradients/parameters.