Chapter 1: Serving LLMs at Scale

LLM Serving Frameworks

Framework	Best For	Key Features
vLLM	High throughput	PagedAttention, continuous batching
TGI	HuggingFace models	Flash attention, tensor parallelism
Triton	Multi-framework	Batching, model ensemble
Ollama	Local deployment	Easy setup, many models

vLLM Deployment

# Start vLLM server
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf",
    tensor_parallel_size=2,  # Use 2 GPUs
    gpu_memory_utilization=0.9,
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512
)

# Batch inference
prompts = ["What is MLOps?", "Explain LLMOps"]
outputs = llm.generate(prompts, sampling_params)

Streaming Responses

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
async def stream_chat(request: ChatRequest):
    async def generate():
        async for token in llm.stream(request.message):
            yield f"data: {token}\n\n"
        yield "data: [DONE]\n\n"
    
    return StreamingResponse(generate(), media_type="text/event-stream")

---

Next: Chapter 2: Inference Optimization