Enterprise RAG doesn't fail on the model. It fails on retrieval nobody measured.

The model is almost never the reason an enterprise retrieval project stalls. It stalls because nobody measured retrieval, and because permissions were bolted on after the search instead of built into it.

That is an unglamorous claim, and it is the one we would defend hardest. Swapping model families is the change teams reach for first, because it is a one-line change and it feels like progress. It usually moves answer quality less than re-ranking your results or fixing a chunk boundary — and it does nothing at all about the failure that gets a project cancelled, which is a confident answer citing a document the person asking was never allowed to open.

Why "which model should we use?" is the wrong first question

Retrieval-augmented generation — RAG, feeding a model text fetched from your own systems at query time — has a clean division of labour. Retrieval decides what the model can possibly know when it answers. Generation decides how that material is phrased. If the relevant passage is not in the context window, no model, at any price, will answer correctly. It will produce something fluent instead.

This is worth being precise about, because "the model hallucinated" gets used for at least three different faults with three different fixes:

Only the third row is partly a generation problem. Note also what is not on this list: fine-tuning. Fine-tuning reliably teaches a model form, format and behaviour. It is a poor mechanism for installing facts, and a worse one for facts that change weekly — which describes most enterprise knowledge. When someone proposes fine-tuning as a fix for wrong answers about current policy, that is usually a retrieval problem wearing a costume.

Measure retrieval on its own, before you look at answers

End-to-end answer quality is a lagging indicator with two variables in it. Score retrieval separately or you cannot tell which half you improved.

You need a golden set: real questions, each labelled with the chunk IDs that genuinely contain the answer. Build it from actual user queries — support tickets, search logs, the questions people currently email a colleague — not from questions you invented while looking at the documents. Questions written by someone reading the source are contaminated by definition; they use the document's vocabulary, and they make retrieval look far better than it is.

Then score two things on every change:

• recall@k — of the passages that genuinely answer the question, how many made it into the top k you pass to the model? This is the ceiling on everything downstream.
• MRR (mean reciprocal rank) or nDCG@k — how high up did the first genuinely relevant passage land? Position matters, because models attend unevenly across a long context.

No framework required:

"""Retrieval scoring over a golden set. No dependencies; Python 3.9+."""
from statistics import mean

def recall_at_k(retrieved_ids, relevant_ids, k):
    """Fraction of the known-relevant chunks that appear in the top k."""
    if not relevant_ids:
        return None  # undefined, not 1.0 — exclude from the average
    hits = set(retrieved_ids[:k]) & set(relevant_ids)
    return len(hits) / len(relevant_ids)

def reciprocal_rank(retrieved_ids, relevant_ids):
    """1 / rank of the first relevant chunk; 0.0 if none in the list."""
    for rank, chunk_id in enumerate(retrieved_ids, start=1):
        if chunk_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def score(golden, retrieve, k=10):
    """golden: [(question, [relevant_chunk_id, ...]), ...]"""
    recalls, rrs = [], []
    for question, relevant in golden:
        retrieved = retrieve(question, k)
        r = recall_at_k(retrieved, relevant, k)
        if r is not None:
            recalls.append(r)
        rrs.append(reciprocal_rank(retrieved, relevant))
    return {"n": len(golden), f"recall@{k}": mean(recalls), "mrr": mean(rrs)}

Two disciplines make the numbers trustworthy. First, report n, and treat a 30-question golden set as a smoke test rather than evidence — small sets produce error bars wide enough to swallow any improvement you are excited about. Second, if you use a model to judge answer quality, mitigate the known biases and say that you did: judges favour longer answers, favour whichever option appears first, and favour outputs from their own family. Randomise presentation order and score against an explicit rubric.

One comparison rule, because it invalidates a lot of published bake-offs: never change the corpus, the chunking, or the k at the same time as the retriever and call the result a retriever comparison. Change one thing.

Permissions belong inside the query

In a consumer chatbot every user can see the whole corpus. In an enterprise, the same question has different correct answers depending on who is asking, and some documents are career-endingly sensitive. This changes the architecture, not just a check at the end.

The tempting design is to retrieve first and filter after. It fails in two directions at once. If you filter the citations but the model already read the text, the content can be restated in the answer even with the link stripped — the leak already happened, in the prompt. And if you filter the results, you have quietly destroyed your top-k: you asked for ten passages, six were removed, and the model now answers from four. Recall collapses for exactly the users with the narrowest access, which tends to mean the newest employees, who ask the most questions.

The design that holds up: access control lives in the index as filterable metadata, and every retrieval is a filtered search executed with the caller's identity. The vector store never returns a passage the caller cannot read, so the model never sees one. That has consequences you should plan for rather than discover:

• ACL changes require re-indexing the affected metadata. Someone changing a SharePoint permission has just invalidated part of your index; if that propagation takes a week, you have a week-long leak.
• Deletion must propagate. A document removed at source and left in the index is now a private copy of something legal asked you to delete.
• Filtered search costs recall. Heavily restricted searches need a larger candidate pool before filtering to return the same k. Budget for it.

Two things worth keeping distinct, because they get conflated in vendor conversations: data security is who can access what, and it is the problem above. Data privacy is what a provider retains and trains on, and it is a contract question, answered by reading the terms — not by anything in your architecture.

And treat retrieved text as untrusted input. If any user, vendor or inbound email can put a document into your corpus, they can put instructions there too, aimed at whoever retrieves it next. Prompt injection has no complete fix; it has layered mitigation — keep retrieved content clearly framed as data rather than instructions, validate outputs, separate privileges so the answering path holds no dangerous tools, and require human approval for consequential actions. The OWASP Top 10 for LLM Applications is the right shared vocabulary here, and it is more useful than inventing your own taxonomy in a design doc.

The boring changes that actually move retrieval

In rough order of return on effort:

Hybrid search. Combine BM25 keyword scoring with dense vector similarity. Enterprise queries are full of tokens that embeddings blur together and exact matching nails: part numbers, error codes, clause references, internal project names, SLA-2.4(b). Dense retrieval alone reliably misses these. Hybrid usually beats either component alone; if your data says otherwise, that is an interesting result and you should show it.

A reranker. Embedding similarity is not relevance — it is a single-vector approximation of topical closeness, computed without the query and document ever meeting. A cross-encoder reranker scores the pair jointly, which lets it express "this passage is about your topic but does not answer your question." Retrieve 50 candidates cheaply, rerank to the 5 you actually send. This is often the single largest quality jump available, and it is a few days of work.

Chunking that respects two constraints. State your chunk size and your embedding model's maximum sequence length together, because the second silently truncates the first — text past the limit is not embedded, and nothing warns you. Then chunk on document structure rather than a fixed character count: headings, clauses, table rows. A chunk that begins mid-sentence under a heading it no longer carries is the direct cause of the "right answer, wrong citation" row in the table above.

Metadata you can filter on. Effective date, document version, jurisdiction, owning team. Half of enterprise retrieval failures are a superseded document beating the current one on similarity, and the fix is a filter, not a better embedding model.

The strongest counterargument: skip retrieval, use a long context window

Context windows are now large enough that stuffing the whole document set into every prompt is a real option, and for small corpora it is often the right one. It deserves a serious answer rather than a dismissal.

Take the arithmetic on its own terms. At a given per-input-token price P, a query that stuffs 200,000 tokens costs 200,000 × P; one that retrieves 6,000 tokens costs 6,000 × P. That is a factor of ~33 in input cost, per query, forever. Latency moves the same way: prefill work grows at least linearly in input tokens, with the attention component growing quadratically, so time-to-first-token degrades noticeably. And KV cache memory scales with context length, which lowers how many concurrent requests a given GPU can serve — a throughput cost that shows up on your bill rather than in the model's spec sheet.

Prompt caching is the honest rebuttal to the cost half: if the same large prefix repeats across queries, cached input is much cheaper than fresh input. But notice what breaks that assumption in exactly our setting — per-user access filtering means each caller's document set differs, so the shared prefix you were counting on fragments per user.

Then there is accuracy, where the intuition is weakest. A context window is a capacity, not a guarantee: published long-context evaluations consistently show retrieval accuracy varying with where in the window the relevant span sits, and degrading as distractor material grows. More irrelevant text in the prompt is not neutral. Retrieval, done well, is a precision mechanism as much as a cost one.

Where long context genuinely wins: corpora that fit comfortably, questions that need synthesis across a whole document rather than a passage from it, and cases where a document is small enough that chunking loses more than it saves. The mature answer is that these compose — retrieve at the document level, then pass whole documents rather than fragments. What long context does not do is remove the need to know which documents to pass.

Agentic RAG multiplies whatever your retrieval does

Multi-step retrieval — a system that searches, reads, reformulates and searches again — is genuinely more capable on multi-hop questions. It is also where unmeasured retrieval becomes unsurvivable, because per-step reliability compounds.

At a 95% success rate per step, ten steps succeed end-to-end about 60% of the time (0.95¹⁰ ≈ 0.599). At 90%, it is 35%. Every step you add is a multiplication, not an addition, and that arithmetic is why most production "agents" are better described as workflows with branching: a fixed sequence, with the model choosing at a small number of well-defined points. Say which one you are building, because the failure modes and the amount of observability you need differ sharply.

Our default is to earn steps rather than assume them. Start with single-hop retrieval, measure it, add a second hop only where the golden set shows single-hop cannot answer, and log every step's query and results so a bad answer can be reconstructed after the fact rather than argued about.

The constraint is usually organisational, not technical

The uncomfortable part of enterprise AI is that the technology is rarely the binding constraint. Absorption is.

A retrieval system makes your documentation's quality legible in a way nothing else does. It will surface the policy that was superseded in 2023 and never deleted, the two conflicting versions of the onboarding process, the spreadsheet that is the real source of truth. None of that is fixable by an engineer. It needs someone with authority over the documents to own freshness, resolve conflicts and decide what is canonical — a role most organisations do not currently have, and the single strongest predictor we see of whether a pilot becomes a system.

Which brings us to ROI claims, and their missing denominator. A credible number includes build cost, ongoing inference cost, the maintenance cost of an index that must track a changing corpus, and an honest account of the baseline being replaced. "Saves 20 minutes per query" means nothing without knowing how often the question is asked and what it cost to answer before. Pilots also flatter themselves: pilot traffic is friendly, curated and asked by people who want it to work. Production traffic is neither, and the gap between the two distributions is where most of the disappointment lives.

What would change our mind

We hold this position because of a pattern, not a proof, so it should be falsifiable. We would revise it if we saw a system shipped with unmeasured retrieval hold its answer quality over six months on a corpus that changes weekly — our claim predicts silent drift that nobody notices until trust is gone. And we would revise the cost argument if per-token pricing and prefill latency fell far enough that stuffing an enterprise-scale corpus per query beat retrieval on both cost and accuracy. The first would surprise us. The second is a question of when, for small corpora, and it is already true for some.

Where to start on Monday

1. Collect 100 real questions from support tickets, search logs and Slack. Not invented ones.
2. Label the chunks that genuinely answer each. This is tedious and it is the whole project.
3. Score your current retrieval: recall@10 and MRR. You now have a baseline, which you did not have before.
4. Add hybrid search and a reranker. Re-score. Change one thing at a time.
5. Move access control into the retrieval filter, and find out how long an ACL change takes to reach your index.
6. Only then argue about which model to generate with.

Steps one and two are the ones teams skip, and skipping them is why the rest becomes guesswork. Everything else on the list is a few days of engineering.

---

Icybee builds and audits retrieval systems for regulated enterprises. If you have a pilot that works in the demo and not in production, the audit usually starts at step three above.