Chapter 1: A RAG feature end to end
Retrieval-augmented generation is the pattern behind most "ask questions about our documents" features: find the relevant passages, put them in front of the model, ask it to answer using only those passages, and show the user where the answer came from. It is the most common AI feature teams are asked to build, and it is a good teaching case because every stage can fail in a specific, diagnosable way.
The architecture
Two pipelines. The offline one runs when documents change; the online one runs per question. Keeping them separate in code and in deployment is the first architectural decision and the one most often skipped.
Stage by stage
1. Ingestion and chunking
Documents are split into chunks small enough to be retrieved precisely and large enough to carry meaning. Typical: 300-800 tokens, with overlap of 10-20 % so a sentence split across a boundary survives in at least one chunk.
Failure point: chunks that cut through tables, code blocks or lists. The retrieved chunk contains half a table and the model fabricates the other half. Use structure-aware splitting (by heading, then by paragraph) and keep tables and code intact.
Decision to record in the spec: chunk size, overlap, and what metadata travels with each chunk (source document, section title, URL, last-modified date). Metadata is what makes citations possible later.
2. Embedding
Each chunk becomes a vector. The embedding model is a dependency like any other: pin its version, because changing it means re-embedding everything, and vectors from two different models are not comparable.
Failure point: mixing versions. Half the index embedded with model A, new documents with model B, retrieval silently degrades. Store the model name with each vector and refuse to query across versions.
3. Vector store and retrieval
pgvector in the PostgreSQL you already run is the right default for most teams; a dedicated vector database is justified when you have tens of millions of vectors or need features Postgres does not have. Retrieve the top k chunks by similarity (k = 5 to 20), optionally re-rank with a cross-encoder, and apply metadata filters (tenant, language, date) in the query, not after.
Failure point: the question and the documents use different vocabulary ("vacation policy" vs. "annual leave entitlement"). Pure vector search handles this better than keyword search but not perfectly. Hybrid retrieval (vector + BM25, merged) is the standard fix.
Failure point: filtering after retrieval. Retrieve top 10, filter by tenant, end up with 2 chunks or - worse - leak another tenant's chunk. Filters belong in the retrieval query.
4. Prompt assembly
The prompt has three parts: instructions, retrieved context, question. The instructions say what to do when the context does not contain the answer, and that matters more than anything else in the prompt:
Answer the question using only the passages below. Quote the passage number you relied on in
square brackets after each claim, like [2]. If the passages do not contain enough information to
answer, say "I could not find this in the documentation" and do not guess.
Passages:
[1] (Source: HR Handbook, section 4.2) ...
[2] (Source: Benefits FAQ) ...
Question: {question}
Failure point: no instruction for the "not found" case, so the model answers from its training data and the user cannot tell the difference. The refusal instruction is non-negotiable.
5. Generation and citations
Call the model with the assembled prompt. Parse the citation markers out of the answer and map them back to chunk metadata so the UI can link to the source.
Failure point: citations that point to passages that do not support the claim. Models will produce a bracket number because they were told to, not because the passage supports the sentence. Chapter 2's evaluation set is where you catch this.
6. API and UI
The API is an ordinary endpoint: POST /ask {question} -> {answer, citations[], latency_ms, trace_id}. Stream the answer if the UI is interactive; users tolerate a 6-second answer that
starts appearing after 400 ms far better than a 3-second blank screen.
The UI shows the answer, the citations as links, and - this is the part teams skip - a way to say "this was wrong". That button is your monitoring (Chapter 2).
The build, using the delivery loop
Intent. Staff can ask questions about the internal handbook and get an answer with links to the source sections. Out of scope: documents outside the handbook, multi-turn conversation.
Spec (abridged). Ten acceptance criteria, including: answers cite at least one passage; a question whose answer is not in the handbook returns the refusal sentence; a question containing another tenant's identifier never retrieves their chunks; p95 latency under 8 s. Twenty example questions with expected source sections (this becomes the evaluation set).
Generation. Offline pipeline first - it is deterministic and fully testable. Then retrieval, tested against the twenty examples (does the expected section appear in the top 5?). Then prompt assembly and generation. Then API, then UI. Each step one agent session with its own failing tests.
Verification. Deterministic parts: unit and integration tests. Retrieval: recall@5 on the twenty examples, must be above 0.9. Generation: Chapter 2.
Release. Behind a flag, to one team, with the "this was wrong" button wired to a log.
Stack decisions, made explicit
| Component | Default choice | When to deviate |
|---|---|---|
| Chunking | Structure-aware, 500 tokens, 15 % overlap | Very short docs (FAQ entries): one entry per chunk |
| Embeddings | A current hosted model, version pinned | Data residency rules: a self-hosted model |
| Store | pgvector | > 20 M vectors, or you need multi-vector / sparse support |
| Retrieval | Hybrid (vector + BM25), k = 10, metadata filters in query | Re-ranker when precision matters more than latency |
| Generation | A mid-size model with a long context | Cost pressure: smaller model + better retrieval usually beats bigger model + worse retrieval |
| API | Streaming endpoint, trace ID on every response | - |
| Observability | Log question, retrieved chunk IDs, answer, latency, cost, user feedback | - |
Summary
- Two pipelines, offline and online; keep them separate.
- Each stage has a characteristic failure: bad chunk boundaries, mixed embedding versions, post-hoc filtering, missing refusal instruction, unsupported citations.
- Build deterministic parts first and test them normally; the model call is one component.
- The spec's example questions become the evaluation set.