Case Study · Enterprise AI · RAG

Case Study: A Production RAG Platform Serving 10k+ Enterprise Documents

Chunking, hybrid retrieval, reranking, evaluation harness, and cost controls behind an enterprise document intelligence platform used daily by knowledge workers.

Role

Lead AI Engineer

Corpus

10k+ docs

Latency

< 1.8s p95

Stack

Python · pgvector · FastAPI

The problem

A knowledge-heavy enterprise had thousands of internal PDFs, wiki pages, and policy documents scattered across systems. Employees spent measurable minutes per task hunting for the paragraph that answered their question. A plain search index gave hits; it didn't give answers.

The approach

We built a RAG platform that grounds an LLM in the customer's own corpus, with citations that link back to the source paragraph. Success required three things production RAG systems usually skimp on: honest evaluation, hybrid retrieval, and unit-economics discipline.

Ingestion and chunking

Every source got a parser tuned to its shape — layout-aware PDF extraction for policy docs, HTML-aware chunking for wiki pages, slide-aware chunking for decks. Chunks carried structural metadata (section path, page number, source URL) so citations could point at the exact anchor, not just the file.

Hybrid retrieval and reranking

Dense vectors alone missed keyword-heavy queries (policy numbers, error codes, acronyms). We fused dense retrieval (embedding model over pgvector) with BM25 lexical retrieval, then reranked the top 50 candidates with a cross-encoder. That ordering — recall-first then precision — consistently beat either signal alone in evaluations.

Evaluation harness

We built a small offline eval set with human-labelled gold answers and citation targets. Every model, prompt, or retrieval change ran against it before merge. LLM-as-judge covered the long tail, but the gold set was the merge gate. Without this, every "improvement" is a vibe check.

Cost and latency controls

A small router picks between a fast/cheap model and a large model based on query complexity. Aggressive prompt caching, batched embeddings, and a semantic cache on frequent questions kept per-query cost well below the naive baseline while holding p95 latency under 1.8 seconds.

Results

Answer quality (measured as top-1 citation accuracy on the gold set) landed north of 85%. Time-to-answer on user-reported tasks dropped from minutes to seconds. Because citations are always visible, users trust the system — they verify when it matters and move on when it doesn't.

Takeaways

Production RAG is a systems problem, not a prompt problem. Chunking, hybrid retrieval, an evaluation harness, and cost controls contribute more to shipped quality than any single model choice. Build the harness first; the model choice becomes a knob.