Quiz: Module 6 - Inference & Deployment

Question 1

What does KV cache store?

1. Previously computed key and value tensors
2. Model weights
3. Token embeddings
4. Gradients

Question 2

What is speculative decoding?

1. Using a small model to draft tokens, verified by large model
2. Random token generation
3. Parallel generation from multiple models
4. Predicting user intent

Question 3

What memory reduction does INT4 quantization provide?

1. 50%
2. 75%
3. 87.5% (to 12.5% of original)
4. 25%

Question 4

What is GPTQ?

1. Post-training quantization method for GPT models
2. A GPT variant
3. A training framework
4. A tokenizer

Question 5

What is vLLM's key innovation?

1. PagedAttention for memory efficiency
2. Better tokenization
3. Improved training
4. Multi-GPU training

Question 6

What does continuous batching improve?

1. Throughput by processing multiple requests simultaneously
2. Model accuracy
3. Training speed
4. Memory usage

Question 7

What is AWQ?

1. Activation-aware Weight Quantization
2. Automatic Weight Quality
3. Average Weight Quantization
4. Advanced Weighted Queries

Question 8

Which library provides on-the-fly quantization?

1. bitsandbytes
2. GPTQ
3. vLLM
4. TensorRT

Question 9

What is TGI?

1. Text Generation Inference - HuggingFace's serving solution
2. Training GPU Interface
3. Token Generation Index
4. Text Grading Intelligence

Question 10

Why use load balancing for LLM serving?

1. Distribute requests across multiple servers for reliability and throughput
2. Reduce model size
3. Improve accuracy
4. Speed up training

Questions 11-30

(Covering advanced optimization topics: flash attention, tensor parallelism for inference, autoscaling strategies, cost optimization, latency vs throughput tradeoffs, A/B testing, monitoring, caching strategies, rate limiting, and deployment patterns)

Question 11

What does Flash Attention optimize?

1. Memory access patterns for attention computation
2. Model architecture
3. Training data
4. Tokenization

Question 12

What is tensor parallelism in inference?

1. Splitting weight matrices across GPUs
2. Running multiple models
3. Batching requests
4. Caching responses

Question 13

When should you use autoscaling?

1. When traffic varies significantly over time
2. For development only
3. Never - fixed capacity is better
4. Only for training

Question 14

What's a key cost optimization strategy?

1. Use smaller/quantized models for simpler queries
2. Always use the largest model
3. Disable caching
4. Run more servers

Question 15

What's the tradeoff with higher batch sizes?

1. Higher throughput but increased latency per request
2. Lower throughput and lower latency
3. No tradeoff
4. Only affects memory

Question 16

What is response caching useful for?

1. Identical or similar repeated queries
2. All queries equally
3. Only first-time queries
4. Training data

Question 17

What is rate limiting?

1. Limiting requests per user/time to prevent overload
2. Limiting model size
3. Limiting output length
4. Limiting training epochs

Question 18

What metric best measures user experience?

1. Time to First Token (TTFT)
2. Total throughput
3. GPU utilization
4. Memory usage

Question 19

What is INT8's typical accuracy impact?

1. Minimal (~1% degradation)
2. Significant (>10%)
3. None at all
4. Improves accuracy

Question 20

What is the benefit of async inference?

1. Non-blocking, allows concurrent processing
2. Better accuracy
3. Simpler code
4. Lower memory

Question 21

What does Triton Inference Server support?

1. Multiple model frameworks and model ensemble
2. Only PyTorch
3. Only TensorFlow
4. Only ONNX

Question 22

When to use streaming responses?

1. Long outputs where users benefit from seeing partial results
2. Short outputs only
3. Never - wait for complete
4. Only for debugging

Question 23

What is model warming?

1. Running inference before production to optimize caches
2. Training with warm data
3. GPU temperature management
4. Gradual learning rate increase

Question 24

What does "tokens per second" measure?

1. Generation speed/throughput
2. Accuracy
3. Memory usage
4. Model size

Question 25

Why use ONNX for deployment?

1. Framework-agnostic format with wide runtime support
2. Only works with PyTorch
3. Better training
4. Smaller models

Question 26

What is prefill vs decode in LLM inference?

1. Prefill processes prompt (parallel), decode generates tokens (sequential)
2. Both are the same
3. Prefill is training, decode is inference
4. Only used in training

Question 27

What does GPU memory limit during inference?

1. Model size and KV cache for context
2. Only model size
3. Only batch size
4. Training gradients

Question 28

What is the benefit of model distillation for deployment?

1. Smaller, faster model that mimics larger one
2. Better accuracy
3. Larger model
4. More parameters

Question 29

What is a health check in LLM deployment?

1. Endpoint to verify service is running correctly
2. Model accuracy test
3. GPU temperature check
4. Training validation

Question 30

What metric indicates GPU efficiency?

1. GPU utilization percentage during inference
2. Model accuracy
3. Number of parameters
4. Training time

---

Passing Score: 24/30 (80%)

← Chapter 3Module 7 →