RAG evaluation under domain shift
A domain-shift evaluation protocol for RAG systems that decomposes retrieval failure, generation failure, context conflict, and abstention behavior.
Whether retrieval and generation failures can be separated reliably under temporal, lexical, authority, and corpus-structure shifts.
- Retrieval/generation error taxonomy
- Domain-shift fixture set
- Faithfulness scoring protocol
- Abstention tests
Generate a domain-shift fixture set with stale, missing, conflicting, and renamed source documents, then score retrieval and generation separately. The protocol should report retrieval precision/recall, faithfulness, abstention quality, and a short error decomposition so the reader can see whether the failure began in retrieval or generation.
RAG systems should be evaluated as two coupled systems, retrieval and generation, because domain shift can degrade context selection, faithfulness, abstention, and answer calibration even when the base model appears unchanged.
RAG failures are often misattributed
A wrong RAG answer can originate in at least four places: the retriever did not find relevant evidence, the ranker surfaced weak evidence, the generator ignored good evidence, or the corpus itself contained stale or conflicting information. Treating the final answer as the only artifact hides the failure source.
Evaluate retrieval and generation separately
| Metric | Layer | Meaning |
|---|---|---|
| Context precision | Retrieval | Retrieved chunks are relevant to the question. |
| Context recall | Retrieval | The needed evidence appears in retrieved context. |
| Faithfulness | Generation | The answer is supported by retrieved evidence. |
| Answer relevance | Generation | The response addresses the user question. |
| Citation support | Generation | Claims map to specific source spans. |
| Abstention quality | System | The system refuses or asks for clarification when evidence is insufficient. |
{
"question_id": "shift-claims-042",
"question": "Can policy X be applied to offshore contractors after the June update?",
"retrieved_context_ids": ["doc-2026-05-policy-x", "doc-2025-11-legacy-policy"],
"gold_context_ids": ["doc-2026-06-policy-x-update"],
"answer_supported": false,
"failure_source": "retrieval_missed_current_policy",
"expected_behavior": "abstain_or_request_current_policy_document"
}The hard cases are temporal, lexical, and policy shifts
- Temporal shift: policies or facts changed after the embedding index was built.
- Lexical shift: users use new terminology that does not match chunk text.
- Structural shift: source documents move from prose to tables, attachments, or ticket threads.
- Authority shift: two sources conflict and one should dominate.
- Scope shift: a question asks for a conclusion that no source supports.
RAG evaluation should be continuous
A RAG benchmark ages as soon as the corpus changes. Production-grade evaluation should therefore run as part of ingestion, index rebuilds, model changes, prompt changes, and retrieval-parameter changes. The key artifact is not a one-time score; it is a regression surface over the retrieval and generation pipeline.
Related research
View allA measurement model for AI deployment readiness
How should teams measure whether an AI system is ready for controlled production deployment?
LLM-as-judge reliability, calibration, and bias
When can LLM judges be trusted as scalable evaluators, and when do they need human calibration?
Benchmark contamination and evaluation leakage
How can AI evaluations avoid overstating performance because the model, prompt, or team has already seen the test?