Chapter 2: Computational Requirements

Calculating GPU Memory and Training Costs

Understanding the computational requirements for fine-tuning is crucial for planning your projects and choosing the right approach.

GPU Memory Components

During training, GPU memory is consumed by four main components:

1. Model Weights

The parameters of the model itself.

2. Gradients

Same size as weights—stores gradient for each parameter.

3. Optimizer States

AdamW stores 2 states (momentum, variance) per parameter.

4. Activations

Intermediate values needed for backpropagation.

Memory Calculation Formula

Basic Formula

Memory (bytes) = Parameters × Precision × Multiplier

Where Multiplier for training:
- Inference: 1
- AdamW training: 4-6 (weights + gradients + 2 optimizer states + activations)

Precision Sizes

Precision	Bytes per Parameter
float32	4
bfloat16	2
float16	2
int8	1
int4	0.5

Full Fine-Tuning Memory Requirements

Example: 7B Parameter Model

# Parameters: 7 billion
parameters = 7_000_000_000

# With float32 (full precision)
weights_fp32 = parameters * 4  # 28 GB
gradients_fp32 = parameters * 4  # 28 GB
optimizer_fp32 = parameters * 4 * 2  # 56 GB (2 states)
# Total: ~112 GB ❌ Too much for most GPUs

# With bfloat16 + mixed precision
weights_bf16 = parameters * 2  # 14 GB
gradients_bf16 = parameters * 2  # 14 GB
optimizer_fp32 = parameters * 4 * 2  # 56 GB (kept in fp32)
# Total: ~84 GB ⚠️ Still requires A100-80GB

# With gradient checkpointing + bf16
# Activations reduced by ~30%
# Total: ~60-70 GB

Memory Requirements Table

Model Size	Full FT (fp32)	Full FT (bf16)	With Grad Ckpt
1B	~16 GB	~12 GB	~8 GB
7B	~112 GB	~84 GB	~60 GB
13B	~208 GB	~156 GB	~110 GB
70B	~1.1 TB	~840 GB	~600 GB

Reducing Memory Requirements

1. Mixed Precision Training

training_args = TrainingArguments(
    bf16=True,  # Use bfloat16
)

Reduces weight and gradient memory by ~50%.

2. Gradient Checkpointing

training_args = TrainingArguments(
    gradient_checkpointing=True,
)
# OR
model.gradient_checkpointing_enable()

Trades compute for memory—recomputes activations during backward pass.

3. Gradient Accumulation

training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    # Effective batch size = 1 * 16 = 16
)

Smaller batches use less memory; accumulate gradients to simulate larger batches.

4. Memory-Efficient Optimizers

# 8-bit Adam (requires bitsandbytes)
training_args = TrainingArguments(
    optim="adamw_8bit",
)

# OR Adafactor (no momentum)
training_args = TrainingArguments(
    optim="adafactor",
)

5. DeepSpeed ZeRO

# deepspeed_config.json
{
    "zero_optimization": {
        "stage": 2,  # Partition gradients and optimizer states
    }
}

# In training
training_args = TrainingArguments(
    deepspeed="deepspeed_config.json",
)

ZeRO stages:

Stage 1: Partition optimizer states
Stage 2: + Partition gradients
Stage 3: + Partition model weights (enables training models larger than GPU memory)

Practical GPU Recommendations

Single GPU Training

GPU	VRAM	Max Model (Full FT)	Max Model (Efficient)
RTX 3090	24 GB	1B	3B
RTX 4090	24 GB	1B	3B
A100-40GB	40 GB	3B	7B
A100-80GB	80 GB	7B	13B
H100	80 GB	7B	13B

Multi-GPU Training

# Automatic distribution with accelerate
from accelerate import Accelerator

accelerator = Accelerator()
model, optimizer, dataloader = accelerator.prepare(
    model, optimizer, dataloader
)

Setup	Combined VRAM	Max Model (Efficient)
2× A100-40GB	80 GB	7-13B
4× A100-80GB	320 GB	30-40B
8× H100	640 GB	70B

Training Time Estimation

Factors Affecting Training Time

Training Time = (Dataset Size × Epochs) / (Batch Size × Throughput)

Throughput Examples (tokens/second)

Model	GPU	Throughput
7B	A100-40GB	~1,000 tokens/s
7B	H100	~2,000 tokens/s
13B	A100-80GB	~500 tokens/s
70B	8× H100	~800 tokens/s

Time Calculation Example

# Dataset: 10,000 examples, 512 tokens each
total_tokens = 10_000 * 512 = 5_120_000

# Training: 3 epochs
total_training_tokens = 5_120_000 * 3 = 15_360_000

# Throughput: 1,000 tokens/second
training_time_seconds = 15_360_000 / 1_000 = 15,360 seconds
training_time_hours = 15_360 / 3_600 = 4.3 hours

Cloud Cost Estimation

GPU Pricing (Approximate)

GPU	On-Demand ($/hr)	Spot ($/hr)
T4	$0.50	$0.15
A10G	$1.50	$0.45
A100-40GB	$4.00	$1.20
A100-80GB	$6.00	$1.80
H100	$8.00	$2.40

Cost Calculation Example

# Training a 7B model
# Time: 4 hours on A100-40GB
# Cost: 4 × $4.00 = $16 (on-demand)
# Cost: 4 × $1.20 = $4.80 (spot)

# With experimentation (5 runs)
# Total: $80 (on-demand) or $24 (spot)

Memory Profiling

Check Current Memory Usage

import torch

# Current allocated memory
allocated = torch.cuda.memory_allocated() / 1024**3
print(f"Allocated: {allocated:.2f} GB")

# Peak memory usage
peak = torch.cuda.max_memory_allocated() / 1024**3
print(f"Peak: {peak:.2f} GB")

# Reset peak stats
torch.cuda.reset_peak_memory_stats()

Estimate Before Training

def estimate_memory(model, batch_size, seq_len, precision="bf16"):
    """Estimate training memory requirements."""
    param_count = sum(p.numel() for p in model.parameters())
    
    # Bytes per parameter based on precision
    bytes_per_param = 2 if precision == "bf16" else 4
    
    # Model weights
    weights_gb = (param_count * bytes_per_param) / 1024**3
    
    # Gradients (same as weights)
    gradients_gb = weights_gb
    
    # Optimizer states (AdamW: 2 states in fp32)
    optimizer_gb = (param_count * 4 * 2) / 1024**3
    
    # Activations (rough estimate)
    activation_gb = (batch_size * seq_len * 4096 * 4) / 1024**3  # Approximate
    
    total_gb = weights_gb + gradients_gb + optimizer_gb + activation_gb
    
    print(f"Weights: {weights_gb:.2f} GB")
    print(f"Gradients: {gradients_gb:.2f} GB")
    print(f"Optimizer: {optimizer_gb:.2f} GB")
    print(f"Activations: ~{activation_gb:.2f} GB")
    print(f"Total: ~{total_gb:.2f} GB")
    
    return total_gb

Optimization Checklist

□ Enable mixed precision (bf16/fp16)
□ Enable gradient checkpointing
□ Use gradient accumulation for larger effective batch sizes
□ Consider 8-bit optimizers
□ Use DeepSpeed for multi-GPU
□ Monitor memory with profiling
□ Use spot instances for cost savings

Key Takeaways

✅ Full fine-tuning requires ~4-6x model size in GPU memory

✅ A 7B model needs 60-80GB for full fine-tuning

✅ Mixed precision and gradient checkpointing significantly reduce memory

✅ Multi-GPU training enables larger models

✅ Cloud costs can be reduced 60-70% with spot instances

✅ Always profile memory before committing to long training runs

Next Steps

Continue to Chapter 3: Catastrophic Forgetting to learn about knowledge retention during fine-tuning.

Quick Check

Before moving on, make sure you can answer:

What are the four main components consuming GPU memory during training?
How much memory does a 7B model require for full fine-tuning with bf16?
What is the purpose of gradient accumulation?

Calculating GPU Memory and Training Costs​

GPU Memory Components​

1. Model Weights​

2. Gradients​

3. Optimizer States​

4. Activations​

Memory Calculation Formula​

Basic Formula​

Precision Sizes​

Full Fine-Tuning Memory Requirements​

Example: 7B Parameter Model​

Memory Requirements Table​

Reducing Memory Requirements​

1. Mixed Precision Training​

2. Gradient Checkpointing​

3. Gradient Accumulation​

4. Memory-Efficient Optimizers​

5. DeepSpeed ZeRO​

Practical GPU Recommendations​

Single GPU Training​

Multi-GPU Training​

Training Time Estimation​

Factors Affecting Training Time​

Throughput Examples (tokens/second)​

Time Calculation Example​

Cloud Cost Estimation​

GPU Pricing (Approximate)​

Cost Calculation Example​

Memory Profiling​

Check Current Memory Usage​

Estimate Before Training​

Optimization Checklist​

Key Takeaways​

Next Steps​

Quick Check​