RAG Recall (Critical Information)
Performance
Overview
The RAG Recall (Critical Information) evaluation measures how completely a RAG model's response covers the ground-truth answer, on questions that target critical information. It shares the recall metric with the general RAG recall evaluation, but its test set is deliberately concentrated on the facts whose incomplete answer carries real legal, financial, safety, health, security, eligibility, or discrimination / bias risk - not mere customer inconvenience.
A low recall score indicates the model omits important information present in the target answer, resulting in incomplete responses. On critical-information questions, an omitted condition, exception, or obligation is the kind of gap most likely to cause real harm.
Metrics
Recall
The fraction of ground-truth answer claims that are correctly answered by the model response (range: 0.0 to 1.0).
Motivation
A model that only answers part of a question may appear correct while leaving the user with an incomplete and potentially misleading picture. In RAG applications where users ask multi-part questions or expect comprehensive summaries, systematic under-coverage is a quality failure distinct from hallucination or unfaithfulness.
This evaluation raises the stakes by focusing the questions on critical information. On a high-risk question, a partial answer that drops a safety caveat, an eligibility exception, a deadline, or a cost can be as harmful as a wrong one - the user acts on an incomplete picture without knowing what was left out. Measuring recall specifically on these questions surfaces the omissions that matter most, and helps attribute them to the retriever, the model's use of context, or an over-selective summarisation strategy.
Methodology
This evaluation measures how completely a model's response covers the ground-truth answer, on a QA set targeting critical information.
- Test Cases: A generator model reads your text documents and writes questions focused on critical facts - those whose incorrect or incomplete answer carries legal, financial, safety, health, security, eligibility, or discrimination / bias risk - each with a ground-truth target answer and supporting quotes.
- Model Response: The evaluated model receives the query, retrieves relevant references from the knowledge base, and produces a response.
- Recall Scoring: The ground-truth target answer is broken down into atomic factual claims. Each target claim is checked against the model response to determine whether it is correctly answered.
- Score Computation: The recall score is calculated as the fraction of target claims that are correctly answered by the model response.
The evaluation focuses on coverage at the claim level. A higher score indicates that a larger fraction of the expected answer is present in the model's response.
Claim extraction and entailment checks are performed by a judge model, which is configurable through the task configuration.
Scoring
Recall Scorer
Examples
High recall - all critical target claims reproduced
Both target claims are reproduced - the eligibility condition (cancelled by the airline) and the 90-day deadline.
Low recall - critical condition omitted
Only the eligibility claim is reproduced; the critical 90-day deadline is omitted, leaving the user able to miss the refund window.
References
[1] Ru D, Qiu L, Hu X, Zhang T, Shi P, Chang S, Jiayang C, Wang C, Sun S, Li H, Zhang Z. Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation. Advances in Neural Information Processing Systems. 2024 Dec 16;37:21999-2027.