Blog / Architecture
RAG Architecture: A Production Blueprint for LLM Retrieval Systems
Retrieval-Augmented Generation (RAG) grounds a language model in your own corpus — reducing hallucinations, enabling citations, and keeping answers current without retraining. This post walks through the full production architecture: ingestion, chunking, embeddings, vector search, reranking, prompt assembly, evaluation, and observability.
1. Why RAG (and when not to use it)
RAG is the right tool when answers must be grounded in a specific corpus — internal docs, contracts, product manuals, medical records, or a knowledge base that changes faster than you can fine-tune. It gives you citations, tenant isolation, and a cheap way to keep the model current.
Skip RAG if the task is pure reasoning over the user's own input (summarization, translation, code transformation) or if the corpus fits comfortably in the model's context window — long-context prompting is simpler and often cheaper at small scale.
2. High-level architecture
A production RAG system has two loops: an offline ingestion pipeline that turns documents into searchable vectors, and an online query pipeline that retrieves, reranks, and generates.
┌─────────────┐ ┌──────────┐ ┌───────────┐ ┌────────────┐
│ Documents │──▶ │ Chunk + │──▶ │ Embed │──▶ │ Vector DB │
│ (PDF, HTML, │ │ Clean │ │ (dense) │ │ + BM25 │
│ DB, wiki) │ └──────────┘ └───────────┘ └─────┬──────┘
└─────────────┘ │
│ (offline)
─────────────────────────────────────────────────────────────────────
│ (online)
┌───────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─▼──────┐
│ User │──▶│ Query │──▶│ Hybrid │──▶│ Rerank │──▶│ Prompt │
│ Query │ │ Rewrite │ │ Retrieve │ │ (cross- │ │ Build │
└───────┘ └──────────┘ │ (kNN+BM25)│ │ encoder) │ └───┬────┘
└──────────┘ └──────────┘ │
▼
┌───────┐ ┌────────────────────┐
│ Eval │◀──│ LLM ─▶ Answer + │
│ + Obs │ │ Citations │
└───────┘ └────────────────────┘3. Ingestion & chunking
Ingestion is where most RAG projects quietly fail. Parse each source with a format-aware loader (Unstructured, PyMuPDF, Docling) so you keep structure — headings, tables, code blocks — instead of a wall of text. Normalize whitespace, strip boilerplate, and preserve section metadata (title, breadcrumb, source URL, permissions).
Chunking strategy
Aim for 300–800 token chunks with 10–20% overlap. Prefer semantic or structural chunking (split on headings, list items, or sentence boundaries) over fixed-size windows. Attach a parent_id so you can retrieve a small chunk for relevance but expand to the parent section at answer time.
{
"id": "doc_842#p12_c03",
"parent_id": "doc_842#p12",
"text": "…semantic unit, 300–800 tokens…",
"title": "Refund Policy › EU Consumers",
"source": "https://…/refunds#eu",
"tenant": "acme-eu",
"acl": ["role:support", "role:legal"],
"updated_at":"2026-06-12T09:14:00Z",
"embedding": [ 0.021, -0.113, … ] // 1024-dim
}4. Embeddings & the vector store
Pick an embedding model that matches your domain and language mix (e.g. text-embedding-3-large, bge-m3, or a domain-tuned variant). Fix the dimensionality up front — changing it later means re-embedding the entire corpus.
For the store, choose based on scale: pgvector for < 10M vectors and strong SQL/ACL needs; Qdrant, Weaviate, or Milvus for larger workloads and native hybrid search; Vespa or OpenSearch when you need rich filtering and BM25 alongside dense vectors. Always combine dense kNN with a lexical index — hybrid retrieval beats either alone.
5. Retrieval & reranking
The online pipeline typically runs three stages:
- Query rewriting — expand acronyms, resolve pronouns from chat history, and (optionally) generate multiple query variants (HyDE, multi-query) to widen recall.
- Hybrid retrieval — fetch top-50 from dense kNN and top-50 from BM25, then fuse with Reciprocal Rank Fusion.
- Reranking — pass the fused 50–100 candidates through a cross-encoder (e.g.
bge-reranker-v2, Cohere Rerank) and keep the top 4–8 for the prompt.
query ─▶ rewrite ─┬─▶ dense kNN (top 50) ─┐
│ ├─▶ RRF fuse ─▶ rerank ─▶ top 6
└─▶ BM25 (top 50) ────┘6. Prompt assembly & generation
Assemble the prompt with a strict template: a system message with rules ("answer only from CONTEXT; if unsure, say so; cite by [id]"), the retrieved chunks with their IDs and source URLs, then the user question. Enforce a token budget — trim from the middle of long chunks, not the ends, and never let context crowd out the instructions.
SYSTEM You are a support assistant. Answer ONLY from CONTEXT. If the answer is not in CONTEXT, reply: "I don't know." Cite sources inline as [<id>]. CONTEXT [doc_842#p12_c03] Refund requests within the EU must be… [doc_301#p04_c01] Digital goods are exempt when… … USER Can I get a refund on a downloaded ebook in Germany?
Stream the response, parse citations into structured references, and verify each citation exists in the retrieved set before returning — a cheap post-check that catches most hallucinated IDs.
7. Evaluation & observability
Treat RAG as a system with two evaluable stages. For retrieval, track Recall@k and MRR against a labeled set of question → gold-chunk pairs. For generation, score faithfulness, answer relevance, and context precision (RAGAS, TruLens, or a bespoke LLM-judge with human spot-checks).
In production, log every request with: query, rewritten query, retrieved IDs and scores, reranked IDs, final prompt token count, model, latency per stage, answer, and user feedback. This log is your regression suite — replay it after every prompt, model, or index change.
8. Common pitfalls
- Chunking too small. Sub-200-token chunks lose context; the model answers from fragments.
- Skipping reranking. kNN alone routinely puts the right chunk at rank 8, outside your top-k.
- Ignoring ACLs. Filter by tenant and role inside the vector query, not after — otherwise you leak documents across users.
- Stale index. Wire ingestion to source webhooks; nightly rebuilds are too slow for changing corpora.
- No eval harness. Without Recall@k and faithfulness scores, every "improvement" is a vibe check.
Want to see this pattern applied end-to-end? See the Enterprise RAG Document Intelligence case study or get in touch.