Chapter 1: Tokenization Algorithms

BPE (Byte Pair Encoding)

Iteratively merges most frequent character pairs:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer

tokenizer = Tokenizer(BPE())
trainer = BpeTrainer(vocab_size=50000, special_tokens=["[PAD]", "[UNK]"])
tokenizer.train(files=["corpus.txt"], trainer=trainer)

WordPiece (Used by BERT)

Similar to BPE but uses likelihood instead of frequency:

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("unbelievable")
# ['un', '##believe', '##able']

SentencePiece (Used by LLaMA, T5)

Language-agnostic, treats text as raw bytes:

import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input='corpus.txt',
    model_prefix='tokenizer',
    vocab_size=32000
)

Comparison

Algorithm	Used By	Key Feature
BPE	GPT-2, GPT-3	Frequency-based merging
WordPiece	BERT	Likelihood-based
SentencePiece	LLaMA, T5	Byte-level, language-agnostic

---

← Module OverviewChapter 2 →