Enterprise RAG-Based Document Intelligence

Every large company I have worked with has the same failure mode around internal knowledge. Contracts live in one drive, HR policies in a wiki, engineering runbooks in a second wiki, security exceptions in a ticket system, and the actual working knowledge lives in a handful of senior people who answer the same questions on Slack five times a week. When I was asked to build a document intelligence platform, the ask on paper was 'a chatbot on our documents.' The ask in reality was much narrower and much harder: give an employee a grounded, cite-backed answer to a specific question in under three seconds, without leaking documents they should not see, and without confidently hallucinating a policy that does not exist.

The moment I heard 'without confidently hallucinating' I knew the design center of the system was not the model. It was the retrieval layer, the evaluation harness, and the refusal behavior. Any modern instruction-tuned LLM will produce a fluent, plausible answer from any context you hand it. The interesting engineering is upstream of that: what context do you hand it, how do you know the context is right, and what do you do when it is not.

The pilot corpus was roughly 1.4 million documents spanning five domains: legal contracts, HR policies, engineering runbooks, security advisories, and a customer-facing knowledge base. Sizes ranged from 200-word FAQ entries to 400-page master service agreements. Formats ranged from clean HTML to scanned PDFs where the OCR quality was, to put it diplomatically, load-bearing. Every domain had its own access rules. Legal contracts were visible to the legal team plus the deal owner. Runbooks were visible to engineering. HR policies were mostly public inside the company but a handful of country-specific policies had to be region-scoped. If a single query leaked a document across those boundaries, the platform would be turned off the next day and I would deserve it.

Latency mattered because the primary interface was a Slack slash command. If a support engineer types a question and waits eight seconds, they context-switch to another tab, and the answer arrives to a distracted human who has already moved on. My internal target was 2.5 seconds end-to-end at p95, including retrieval, reranking, generation, and streaming the first token. That target drove almost every architectural decision downstream.

I wrote down the constraints on day one because they doubled as tripwires. Any decision that violated one of them was wrong by definition, no matter how elegant the code looked. The constraints were: strict tenant and role isolation at the retrieval layer, not just at the UI; deterministic citations tied to a specific chunk with a stable identifier that survived reindexing; refusal on low-confidence retrieval instead of best-effort generation; observable end-to-end so I could debug a bad answer six weeks later; and no vendor lock at the embedding layer, because I already knew we would swap embedding models at least twice during the first year.

The non-goals were just as important. I explicitly refused to build a general chat interface. The system answers one question at a time and does not maintain long conversational memory. I refused to expose raw model output when retrieval confidence was below threshold. I refused to summarize documents users had not opened themselves in the UI, because a summary is still a citation-free assertion and I did not want to be in the business of defending summaries. And I refused, for the first six months, to add tools or agentic behavior. Retrieval-augmented generation with strict grounding is already a system with a lot of failure modes. Adding tool calls on top of an unreliable retrieval layer is how you ship a demo that looks great in a screenshot and cannot be operated in production.

The last constraint I want to name explicitly is cost per answered question. I set an internal budget of roughly $0.02 per query at steady state, inclusive of embeddings, storage, reranking, and generation. That budget forced me to think carefully about which model handled which step. The most expensive model in the stack should be doing the smallest amount of work, and only after cheaper components have narrowed the problem down.

The final architecture has five stages in the hot path and three offline pipelines feeding them. In the hot path, a query hits an ingress FastAPI service that resolves the user's tenant, role, and region into a set of access predicates. Those predicates are pushed down as metadata filters into two parallel retrievers: a dense retriever over a Pinecone index and a sparse retriever over an OpenSearch BM25 index. Each retriever returns its top 50. I merge the two lists using reciprocal rank fusion, then hand the fused top 40 to a cross-encoder reranker that scores each candidate against the query. I keep the top 8, assemble a prompt with structured citations, and call the generator with a strict system prompt that requires either a cited answer or an explicit refusal. Streaming starts as soon as the first token is available.

The offline pipelines handle ingestion, evaluation, and drift monitoring. Ingestion is document-type-aware: contracts go through a clause-aware chunker that respects legal section boundaries, policies go through a heading-aware chunker that keeps definitions attached to the term they define, and runbooks go through a code-block-aware chunker that never splits a fenced code sample. Every chunk carries a stable identifier derived from a content hash plus a document version, so citations point to a specific chunk revision, not a moving target. Evaluation runs nightly against a curated set of 900 question-answer pairs with expected citations, and drift monitoring compares retrieval score distributions week over week.

The choice of hybrid retrieval over pure dense retrieval was the single most impactful architectural decision I made, and I want to be honest that I did not start there. My first implementation was pure dense with a good embedding model, and the recall was fine on paraphrased questions but embarrassing on questions that contained specific identifiers, product SKUs, error codes, and clause numbers. Users asking 'what does section 12.4 of the MSA say about termination for convenience' were getting semantically similar clauses from other contracts instead of the exact section from the exact document. BM25 solved that class of query overnight, because a rare token like a section number carries enormous discriminative weight in a sparse index and almost none in a dense one. The fusion step is where you get both. I use reciprocal rank fusion with a k of 60, which is close to the value the original paper recommends, and I have not found a domain where tuning it further has been worth the effort.

The reranker is a cross-encoder I fine-tuned on our own labeled pairs. Off-the-shelf rerankers were fine on general text but had a noticeable blind spot on domain jargon, especially in the legal and clinical corpora. Fine-tuning on 12,000 labeled query-chunk pairs, roughly half positives and half hard negatives mined from the retriever, moved nDCG at 10 by about 6 points on our held-out set. The reranker is the most latency-expensive component in the hot path after the LLM, so I run it on a batched GPU service with a strict 120 ms budget and a fallback to raw fusion scores if the budget is exceeded. That fallback has fired maybe twice in production, both times during deploys.

The first tradeoff I want to talk about is chunk size and overlap. There is a folk belief in RAG that you should chunk at 512 tokens with 50-token overlap because that is what everyone else does. I tried that and it was mediocre for us. What actually worked was making chunking a first-class per-document-type decision. Contracts chunk on clause boundaries with no overlap, because clauses are self-contained legal units and overlap causes duplicate citations that confuse the reader. Policies chunk on heading boundaries with a small header prefix injected into each chunk, so a chunk about 'accrual' still carries the parent heading 'Paid Time Off' as context. Wiki and knowledge-base articles chunk at roughly 400 tokens with 80-token overlap because they are conversational and paragraphs bleed into each other. Runbooks preserve code blocks intact regardless of size, because splitting a shell script in half is the fastest way to give an engineer an answer that will destroy their weekend.

The second tradeoff was embedding model choice. I evaluated four candidates on our corpus using a home-grown benchmark of 900 queries with graded relevance. The differences between the top two were within noise on the paraphrase-heavy queries, but one model was consistently better on multi-hop questions that required stitching context across two chunks. I picked that one and locked the choice for six months to avoid churn in the index. When I re-evaluated later, a newer model was clearly better and we did the full reindex, which took about 40 hours of GPU time and cost roughly $600. Budgeting for a reindex twice a year is the honest way to plan; pretending your embedding choice is permanent is not.

The third tradeoff was the generator model. My default is a mid-tier instruction-tuned model with a strict system prompt and temperature at 0.2. For questions the router flags as high-stakes, defined by the presence of certain compliance keywords or by an explicit user opt-in, the request is routed to a stronger model with a longer context window. About 4% of queries hit the stronger model, which keeps the cost profile close to the cheaper model while giving the hardest questions the horsepower they need. I resisted the temptation to route every query to the strongest available model. The strongest model is also the slowest, and on well-retrieved context the quality difference is small enough that it does not justify the latency or the cost.

The fourth tradeoff was refusal behavior. Every RAG system I have seen fails in the same way at first: when retrieval is bad, the model still generates something because it is designed to be helpful. I forced the opposite behavior by adding a confidence gate between the reranker and the generator. If the reranker's top score is below a learned threshold, or if the top three scores are within a narrow band that indicates the retriever is guessing, the system returns a structured refusal that says 'I do not have a confident answer for this question in the connected documents' and offers to open a ticket with a human. The threshold was picked by looking at the distribution of scores on the eval set and choosing the value that maximized F1 on the 'should have refused' subset. In production, roughly 7% of queries hit refusal, and the follow-up survey on those refusals is where most of my content-gap insights come from.

The fifth tradeoff was citation format. I went through three iterations. Version one had numeric citations at the end of paragraphs, which looked clean but obscured which sentence came from which source. Version two had inline bracketed citations after each sentence, which was more honest but visually noisy. Version three, which is what runs today, splits the answer into short claim-sized paragraphs and puts a single citation card at the end of each claim, with the document title, section, and a preview snippet on hover. Users could actually verify answers against source material for the first time. That version-three change moved trust survey scores more than any model swap I ever did.

The ingestion pipeline is a set of Python workers coordinated by a lightweight job queue. When a document lands in the source system, an event fires and a worker pulls the document, detects the type, runs the type-specific chunker, generates embeddings in batches of 64, upserts to Pinecone with metadata, and indexes the sparse representation into OpenSearch. The metadata carries tenant, role visibility, region, document type, source system, document version, and the stable chunk identifier. Every write is idempotent on the chunk identifier so a reingest is safe. This idempotency has saved me at least four times when a schema change forced a partial reindex.

The tenant isolation implementation is the piece I am most careful about. I do not rely on filtering in the application layer alone. Every query carries an access-predicate object that is translated into both a Pinecone metadata filter and an OpenSearch query clause before the query is issued. The application layer cannot even see chunks it should not see. I audited this with a red-team script that generates queries as user A but injects user B's tenant into the payload at various layers, verifying that the retriever never returns cross-tenant results even when the application layer is misconfigured. I run that audit on every deploy.

The retrieval layer is fronted by a small FastAPI service with async endpoints. Concurrency is bounded per user to prevent one runaway process from starving the pool, and every downstream call has a strict timeout with a circuit breaker. If Pinecone is slow, the sparse retriever's results still make it through, and the system degrades gracefully to sparse-only rather than failing the request. If the reranker is unavailable, fusion scores are used directly. If the generator is unavailable, the system returns the top three chunks with a note explaining that answer synthesis is offline. That degradation ladder is written down in the runbook and rehearsed in game days.

The prompt is deliberately boring. It tells the model that it is a document-grounded assistant, that it must ground every claim in the provided context, that it must use the citation format the system expects, and that it must refuse when the context does not answer the question. I have not found a clever prompt that beats a clear one. What has helped is including a few short in-prompt examples of good refusals, which makes the model much less likely to invent a plausible-sounding answer when the context is thin. In-prompt examples of good citations also help; the model mimics the pattern more reliably than when I only describe the pattern in words.

Observability is boring in a good way. Every query has a trace with the query text, the resolved access predicates, the retrieved chunk ids and scores from both retrievers, the fusion output, the reranker scores, the final chunks, the prompt, the model response, and the citations produced. Traces are sampled at 100% for the first two weeks of a new deploy and 10% at steady state, with automatic full-sampling for refusals and for queries where the user marks the answer as unhelpful. Being able to open a specific trace and see exactly what the retrieval layer did is the single most useful debugging tool I have.

The first serious incident was a silent quality regression after a Pinecone SDK upgrade changed the default distance metric behavior in a subtle way. Retrieval quality dropped roughly 8 points on the eval set overnight, and nobody would have caught it if the nightly eval had not been running. That incident is the reason I now pin every dependency in the retrieval path and gate deploys on the eval delta, not just on unit tests. If the delta on the golden set is worse than a small negative threshold, the deploy is blocked.

The second incident was a subtler one. A team migrated a large batch of contracts into the corpus without informing me, and the contracts were in a scanned PDF format with OCR quality much worse than the rest of the corpus. The chunker happily produced garbage chunks, the embeddings were noisy, and refusal rates on questions about those contracts shot up. Users complained that the system 'used to work.' The fix was two things. First, an ingestion-time OCR-quality gate that flags documents with a confidence score below a threshold and routes them to a review queue rather than the live index. Second, a per-source-batch retrieval-quality dashboard so a bad ingest is visible within an hour instead of a week.

The third incident was a legitimate leak that never reached a user because of the layered filtering, but it should not have happened at any layer. A misconfigured group in the source system briefly gave a broad internal role visibility into a restricted contract set. The access predicates were correct in the retrieval layer, but they trusted the source system's group membership. I added a second layer of assertion: for a small set of highly sensitive document types, the retrieval layer maintains its own allowlist that is validated out of band, so a misconfiguration upstream cannot open access. Defense in depth is not a slogan, it is a Tuesday morning at 3 a.m. when you are grateful for the second layer.

The fourth ongoing lesson is about the eval set itself. It rots. Questions that were representative six months ago drift as the corpus and the users evolve. I now rotate 10% of the eval set every quarter, sampling from real production queries that hit refusal or got a thumbs-down, and I retire questions that everyone in the system has memorized. The eval set is a living asset, not a fixed benchmark.

At the end of the pilot, the platform was handling roughly 4,000 queries per day across the pilot teams, with a p95 end-to-end latency of 2.1 seconds and a p99 of 3.4 seconds. The evaluation harness reported an answer-with-correct-citation rate of 84% on the held-out set, up from 61% at the end of the pure-dense prototype. Refusal precision, measured as the fraction of refusals that were correct refusals, was 91%. Research time per query, measured through a diary study on the legal and support pilot teams, dropped roughly 60% versus the pre-platform baseline, which was the outcome the business had asked for.

Cost per answered query landed at $0.018 at steady state, comfortably under the $0.02 budget. About 62% of the cost was generation, 21% was embeddings and vector storage amortized, 12% was reranking, and the remainder was infrastructure overhead. The largest single cost win was moving to a cheaper default generator and reserving the stronger model for the 4% of high-stakes queries, which saved roughly 35% on generation without a measurable drop in answer quality.

The soft outcome I care about most is that the legal and HR teams stopped treating the system as a novelty and started treating it as a tool. When a system moves from 'let me try that in the AI thing' to 'the answer is in Docs, section 4' in the way employees speak, that is when you know it has crossed the trust threshold.

If I were starting over today, I would build the evaluation harness before I built the retriever. I built them in the opposite order and paid for it in wasted iteration. A crappy retriever with a great eval loop converges quickly. A great retriever with no eval loop drifts silently. Order matters.

I would also have invested earlier in a self-serve corpus-hygiene dashboard for document owners. A lot of the quality issues I hit in year one were not model or retrieval issues; they were corpus issues. Duplicates, stale documents, contradictory policies, and outright wrong information were the root cause of a surprising fraction of bad answers. Giving document owners a view into 'these are your documents, this is how often they are retrieved, these are the questions users ask about them' turned them into partners in quality rather than upstream sources of noise.

Finally, I would have resisted the pressure to add agentic behavior even more strongly than I already did. There were moments where the temptation to add 'and then it can also file the ticket for you' was almost overwhelming, and every time I have seen that path taken prematurely, the retrieval quality regresses because the team stops paying attention to it in favor of the shinier problem. Grounding first, actions later. Always.