Lessons 1-2: chunking and embeddings
Naive fixed-size chunking is the number one cause of bad RAG. Prefer semantic chunking (paragraphs, markdown sections, document structure) with a 10-15% overlap.
On technical documents, add a metadata layer (chapter title, hierarchy, source) — this metadata boosts hybrid retrieval (dense + BM25) by 20-30%.
On embeddings: don't switch models every month. Cohere multilingual v3 and OpenAI text-embedding-3-large are solid and stable. Keep the same model for the entire lifetime of an index to avoid costly reindexing.
Lesson 3: the reranker is non-negotiable
Two-stage retrieval (embeddings → top 50, then cross-encoder reranker → top 5) always outperforms single-stage retrieval. Marginal additional cost, transformed answer quality.
We use Cohere Rerank 3 or self-hosted BGE-Reranker-v2 depending on budget. On the legal use case (200 k documents), the reranker took recall@5 from 71% to 89%.
Cost tip: for simple queries you can skip the reranker and use embeddings only. For complex queries, both stages are mandatory.
Lesson 4: citations are mandatory
A RAG answer without citations is not a RAG answer. It's an opinion. Every output must be sourced: the model is prompted to cite [doc_id, page, section] for each claim.
UX: every citation is clickable and opens the source document with the passage highlighted. That is what turns a doubtful assistant into a professional tool — the user can verify.
On the API side: require structured output from the model (JSON with `answer` + `citations`). OpenAI function calling or Anthropic tool use handle this natively. Robustness is total.
Lesson 5: eval, eval, eval
Without an evaluation dataset, you are blind. Before any production launch, build 200-500 reference questions with the expected answer (written by a domain expert).
Metrics to track: recall@5 (retrieval quality), citation accuracy (are the citations correct?), hallucination rate (unsourced claims), p95 latency.
Every change (prompt, model, retrieval) goes through automatic evaluation on this dataset. Below the threshold, no deploy. Above it, compare with an A/B test in production.
Tools: Langfuse for tracing, Ragas for automatic metrics, Arize for monitoring. The combo is formidable.
Lesson 6: cost, cache, fallback
The cost of a RAG query ranges from €0.02 (GPT-4o-mini + embeddings) to €0.40 (Claude Opus + reranker). Across thousands of queries per day, the gap adds up.
Semantic cache: if a query is similar (cosine > 0.95) to a recent one, serve the cached answer. Typically cuts 30-40% of model volume.
Tiered fallback: for simple queries (factual lookup), use GPT-4o-mini. For complex queries (analysis, synthesis), switch to Claude Sonnet. For premium cases, Opus. A 60% cost reduction with no perceived quality loss.
Lesson 7: governance above all
GDPR: analyse which data enters the vector index. If PII is involved: minimisation, anonymisation, lawful basis, right to erasure (reindexing).
Logs: trace every query (user, prompt, answer, model, latency, cost). Anonymise the prompt after the fact if it contains PII. Retain for 12 months.
Audit: every access to a sensitive document is traced. Every answer export is traced. A RAG system is not ChatGPT — it's an information system, not a toy.