Chapter 2: Quantization (INT8, INT4)
Why Quantize?
Reduce memory and increase speed by using lower precision:
| Precision | Bits | Memory | Quality |
|---|---|---|---|
| FP32 | 32 | 100% | 100% |
| FP16 | 16 | 50% | ~100% |
| INT8 | 8 | 25% | ~99% |
| INT4 | 4 | 12.5% | ~95-98% |
GPTQ
Post-training quantization optimized for GPT-style models:
from transformers import AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM
model = AutoGPTQForCausalLM.from_quantized(
"TheBloke/Llama-2-7B-GPTQ",
device="cuda:0"
)
AWQ (Activation-aware Weight Quantization)
Preserves important weights:
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized(
"TheBloke/Llama-2-7B-AWQ",
fuse_layers=True
)
bitsandbytes
On-the-fly quantization:
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b",
quantization_config=bnb_config
)