RAG in production: 18 months of field lessons

In February 2025, we reviewed seven solid lessons from putting RAG into production: semantic chunking, rerankers, citations, evaluation, governance. Twelve months later, the field has moved again. Three changes now dominate, and none of them was mature a year ago.

First shift: multimodal RAG has become viable. Client corpora are no longer text-only — they contain financial tables, technical diagrams, reporting charts, screenshots of internal tools. Vision models (Claude Sonnet 4.5, GPT-4o, Gemini 2.0 Flash) read these elements directly, without a degrading pre-OCR step. On a banking compliance project, 38% of the useful documents were tables or charts — without multimodal capabilities, more than a third of the corpus was invisible.

Second shift: structured outputs have reached production maturity. OpenAI's strict JSON schemas (`response_format: json_schema`) and Anthropic's tool use now guarantee an output conforming to a schema, character for character. We no longer write post-processing regexes to extract `answer` and `citations`: it is native, contractually guaranteed, and it fails cleanly when the model fails.

Third shift: agentic RAG has replaced naive single-hop. Rather than a single retrieval followed by generation, the model iterates: it retrieves, reads, decides whether it has enough, otherwise reformulates and retrieves again (multi-hop). On multi-hop questions — "compare the 2023 and 2024 HR policies on remote work" — the agentic pattern outperforms classic RAG by 20 to 30 faithfulness points.

In 2025, we said: "build a dataset of 200 to 500 questions". That is still true. What changed is the ability to score every answer automatically. LLM-as-judge — using a model to evaluate another model's output against defined criteria — has moved from research paper to CI pipeline.

Three frameworks dominate late 2025. Ragas v0.2 exposes solid metrics: faithfulness (is the answer grounded in the retrieved contexts?), answer relevancy, context recall/precision. DeepEval offers a convenient Pytest-native integration for Python teams. Promptfoo is our choice for cross-model non-regression testing: we plug the same dataset into GPT-4o, Claude Sonnet 4.5, and Llama 3.3 70B, and compare the deltas in a single table.

The real cost of continuous evaluation still surprises us. On a legal project (480 reference questions, scored by Claude Sonnet 4.5 as judge), each full evaluation pass costs 11 to 14 euros. Run on every PR that touches the prompt, the chunking, or the model, that is 30 to 80 euros per week — negligible. The real hidden cost is gold-standard maintenance: 5 to 8% of the questions must be re-annotated every quarter because the business context evolves.

The classic LLM-as-judge traps: length bias (a judge scores long answers higher) and position bias (does it rate the first answer presented more favourably). We mitigate by randomising the order, constraining lengths, and systematically cross-checking the judge's score against a measured recall@5 on retrieval. If the two metrics diverge, we investigate. A judge alone is never enough.

Prompt caching — launched by Anthropic on Claude in August 2024 and by OpenAI on GPT-4o in October 2024 — is the most significant cost reduction we have seen in two years. Concretely: any identical prompt prefix across two requests is cached on the provider side, and billed at 10 to 50% of the normal input rate for 5 minutes (Anthropic) to 1 hour (OpenAI).

On a RAG system, the system prompt, the tool documentation, and the citation list form a stable prefix of 2,000 to 6,000 tokens. With a consistent cache hit, input cost drops by 50 to 90%. On an internal support project (12,000 requests/month, 4,800-token system prompt), moving to caching cut the Claude bill from 1,180 euros to 320 euros monthly — with zero change on the user side.

The semantic cache remains relevant at another layer. On near-duplicate questions (cosine > 0.95 on the query embedding), we serve the previous answer without calling the model again. Our typical hit rates: 35 to 45% on internal chatbots (where employees often ask the same questions), 12 to 18% on customer-facing assistants (where diversity is higher). Tools tested: Redis with `redisvl`, self-hosted GPTCache, or managed Redis Vector Search.

The third lever is delegation to small models. Claude Haiku 4, GPT-4o-mini, and self-hosted Llama 3.3 70B handle 70 to 85% of traffic on our projects: query classification, short summarisation, reformulation, entity extraction. The large model (Sonnet 4.5, GPT-4o) only steps in for long-form final generation and multi-hop reasoning. Average cost per complete RAG request in 2026: 0.018 euro, versus 0.07 euro at equivalent configuration in February 2025.

For two years, the dominant narrative announced the death of BM25 in favour of embeddings alone. 2025 practice buried it. On technical, legal, or medical corpora — anywhere identifiers, reference numbers, and precise acronyms matter — BM25 alone often beats dense alone, and fusing the two beats each taken in isolation.

The reference trio in 2026: BM25 + dense embeddings + cross-encoder reranker. Embeddings capture semantic similarity ("how to suspend an account" ≈ "user blocking procedure"); BM25 captures exact lexical matching ("article L.1235-7" must surface even when the user types that exact code); the reranker reorders the top 50 candidates with a costlier but more precise attentional model.

For sparse-dense fusion, two serious options have emerged. SPLADE produces learned sparse embeddings that combine natively with a dense vector in a single index. ColBERT2 keeps a distinct vector per token and performs late interaction at query time — precise but storage-hungry (10 to 30 times bulkier than a single dense embedding). We use ColBERT2 on small, critical corpora (< 50,000 documents, legal), and BM25+dense+reranker everywhere else.

For reranking, in 2026 we oscillate between Cohere Rerank 3 (managed, 2 euros per 1,000 searches) and self-hosted BGE-Reranker-v2 (shared GPU, zero marginal cost after amortisation). On the legal project cited above, recall@5 rises from 71% (dense alone) to 89% (dense + rerank) — a figure that is stable versus 2025, confirming that a reranker's structural gain remains massive whatever the state of the art in embeddings.

Three recent incidents deserve a public account, because they are reproducible on any team. First case: a silent chunking change. We had moved from paragraph-based splitting to fixed 512-token overlap splitting to "simplify". No alert went off, but over two weeks the "I don't know" answer rate climbed from 4% to 11%. Cause: the chunks were cutting through the middle of paragraphs containing the critical definitions. Diagnosed by recall@5 drift on the evaluation dataset. Repaired in under 24 hours, but two weeks of user degradation.

Second case: a model version upgrade that introduced subtle hallucinations. Migration from Claude Sonnet 3.5 to Sonnet 4 — spectacular overall quality gains, but on 8% of niche technical questions, the model started inventing regulatory reference numbers instead of admitting absence. The LLM-as-judge in CI caught them before production. Without continuous evaluation, users would have reported the error — far too late.

Third case: a reranker outage that degraded quality without triggering an alert. Cohere Rerank 3 was failing intermittently with 503s, and our fallback code served the raw dense results without telling anyone. P95 latency was fine, answer rate was fine, but recall@5 was silently falling from 89% to 71%. We added a dedicated probe: reranker success rate over 5 minutes, alert if < 95%. Since then, no reranker goes down without our knowing.

Common lesson: a RAG system often degrades without a hard failure. The metrics to watch are not only latency or HTTP error rate — you must monitor recall@5 continuously (via a sample of annotated queries), the abstention rate ("I don't know"), the distribution of citation scores, and any drift > 5% over 24 hours.

If we were starting a new enterprise RAG project in January 2026, here is the stack we would lay down. Vector store: Qdrant for performance and rich metadata filters, or pgvector when the client wants to stay on a single Postgres database and accepts a slight latency penalty. Both are solid; the choice depends on the existing infrastructure.

Embeddings: BGE-M3 (multilingual, natively sparse+dense+ColBERT, self-hosted on a single GPU) has become our French-language default. For strictly English corpora, OpenAI text-embedding-3-large remains excellent. We avoid non-reproducible proprietary embeddings — re-indexing a 500,000-document corpus is expensive in time, not in money.

Rerank: Cohere Rerank 3 managed to start fast, self-hosted BGE-Reranker-v2 once volume justifies a dedicated GPU. Generation model: Claude Sonnet 4.5 for reasoning-heavy cases and long technical corpora; GPT-4o for speed and multimodal coverage; Gemini 2.0 Flash when throughput and cost dominate. Haiku 4 or GPT-4o-mini for routing and classification tasks.

Observability and evaluation: Langfuse for tracing every request (prompt, retrieved context, answer, latency, cost), Promptfoo in CI for non-regression, PromptLayer for prompt versioning and runtime A/B. The trio costs 0 euros in licences (all open-source, self-hostable) and covers 90% of needs. On a recent project, this stack handled 14 million monthly requests on 4,800 euros per month of infrastructure — an order of magnitude below what we reported in 2025.

Three weak signals could become dominant in 2026. First signal: long contexts (1 to 2 million tokens on Gemini, 400k on Claude, 200k on GPT-4o) call chunking itself into question — why split if the whole corpus fits in context? On small, stable corpora (< 200,000 relevant tokens), "long-context RAG" (no chunking, minimal retrieval, large model) becomes competitive. But cost and latency remain prohibitive beyond that.

Second signal: agent tooling — Anthropic's MCP (Model Context Protocol), standardised function calling — is making agentic RAG mundane. The "retrieve, verify, use another tool if needed" pattern is no longer an R&D project; it is a documented primitive. Over the first quarter of 2026, we are deploying two agentic RAGs in client production with this pattern.

Third signal: regulation is hardening the frame. The AI Act is entering into application progressively, traceability requirements for GPAI systems are thickening, and client audits now demand proof of continuous evaluation. A clean evaluation and logging infrastructure — Langfuse, Promptfoo, timestamped traces — is no longer an engineer's luxury; it is a regulatory asset. Anticipate it from the design stage.