RAG Citation Quality
Performance
Overview
The RAG Citation Quality evaluation measures whether a RAG model cites the correct source documents and passages for its response. A low score indicates the model cites the wrong documents, cites passages that do not support its claims, or omits sources it should have cited.
Metrics
Context Recall
The fraction of ground-truth reference passages that are matched by at least one response citation (range: 0.0 to 1.0).
Context Precision
The fraction of response citations whose cited passage matches at least one ground-truth reference (range: 0.0 to 1.0).
Context F1
The harmonic mean of Context Recall and Context Precision (range: 0.0 to 1.0).
Source Recall
The fraction of expected source documents that appear in the response citations (range: 0.0 to 1.0).
Source Precision
The fraction of response-cited source documents that are among the expected sources (range: 0.0 to 1.0).
Source F1
The harmonic mean of Source Recall and Source Precision (range: 0.0 to 1.0).
Motivation
The citations provided in the RAG system's response give us insights into how the answer was conceived. Comparing the citations from the response with the set of expected citations can help with understanding why the model might not be performing as expected. This evaluation measures citation quality at two levels:
- Source level: Did the model cite the right documents? A model may retrieve irrelevant documents (low source precision) or miss key documents entirely (low source recall).
- Passage level: Did the model cite the right passages within those documents? Citing the correct document is necessary but not sufficient — a model can cite the right source yet ground its answer in a passage that does not support the claim (low context precision), or fail to cite the specific passages that do (low context recall).
Understanding both dimensions helps diagnose whether a model is retrieving the wrong documents, citing the wrong passages of the right documents, or citing the right passages but still coming to the wrong conclusions.
Methodology
This evaluation measures whether a model's cited sources and passages match the ground-truth references and expected sources for the query.
- Test Cases: This evaluation relies on test cases that consist of a user query that can be answered using the RAG knowledge base, together with ground-truth reference evidence passages and expected source documents.
- Model Response: The evaluated model receives the query, retrieves relevant documents from the knowledge base, and produces a response with citations referencing the passages and documents used.
- Scoring: Each ground-truth reference is checked against the response's citations to determine whether its passage was matched (recall), and each response citation is checked against the ground-truth references to determine whether it is correct (precision). The same comparison is performed at the document level to compute source recall and precision. Context and source recall/precision/F1 scores are computed as the fraction of matched references, citations, and documents, respectively.
Passages are compared using the match_method configured for the evaluation.
The available methods are:
substring: Each part of the response passage (split on ellipsis markers…or...) must appear inside the ground-truth passage, or the full ground-truth passage must appear inside the response passage. Fast and exact, but sensitive to minor wording differences.ngram(default): Token n-gram overlap between the two passages must reach the configuredmatch_threshold. Tolerant of minor paraphrasing and truncation. Note that the overlap is computed for the shorter passage, meaning that a short and a long passage can still match, as long as most of the short passage is contained in the long one.embeddings: OpenAI embeddings are used to compute cosine similarity between the passages. A match is declared when the similarity reaches the configuredmatch_threshold. Requires theopenai_api_keyto be set in the task configuration.llm: An OpenAI model judges whether both passages refer to the same part of the source document, regardless of wording. The most flexible method but slowest and most expensive. Requires theopenai_api_keyto be set in the task configuration.
Note that passage matching is deliberately tolerant in one edge case: if the
ground-truth reference has no content (empty passage), it is considered matched
by any response citation that cites the same document.
Scoring
Context Recall
key: context_recall
label: Context Recall
entries:
- value: 1.0
color: green
explanation: All ground-truth passages are each matched by at least one response citation - the model correctly cited all expected supporting passages.
- value: 0.5
color: medium
explanation: Half of the ground-truth passages are matched by at least one response citation - the model cited some but not all expected supporting passages.
- value: 0.0
color: red
explanation: None of the ground-truth passages are matched by any response citation - the model failed to cite any of the expected supporting passages.
Context Precision
key: context_precision
label: Context Precision
entries:
- value: 1.0
color: green
explanation: All response citations' passages each match at least one ground-truth reference - every citation is correct and refers to an expected supporting passage.
- value: 0.5
color: medium
explanation: Half of the response citations' passages match a ground-truth reference - some citations are correct while others are spurious or refer to an unexpected passage.
- value: 0.0
color: red
explanation: None of the response citations' passages match any ground-truth reference - all citations are spurious or refer to an unexpected passage.
Context F1
The harmonic mean of Context Recall and Context Precision.
Source Recall
key: source_recall
label: Source Recall
entries:
- value: 1.0
color: green
explanation: The expected source documents appear in at least one response citation each - the model correctly cited all expected sources.
- value: 0.5
color: medium
explanation: Half of the expected source documents appear in a response citation - the model cited some expected sources but omitted others.
- value: 0.0
color: red
explanation: The expected source documents do not appear in any response citation - the model failed to cite any of the expected sources.
Source Precision
key: source_precision
label: Source Precision
entries:
- value: 1.0
color: green
explanation: The cited source documents are all among the expected sources - these citations refer to expected source documents only.
- value: 0.5
color: medium
explanation: Half of the cited source documents are among the expected sources - some citations refer to correct sources while others are unexpected or irrelevant.
- value: 0.0
color: red
explanation: The cited source documents are not among the expected sources - these citations refer to unexpected or irrelevant source documents.
Source F1
The harmonic mean of Source Recall and Source Precision.
Examples
High citation quality - correct source and passage cited
| Citation | Evidence Passage |
|---|---|
| Remote Work Policy.pdf | Employees shall submit a remote work request at least five (5) business days prior to the requested start date. |
Remote Work Policy.pdf
- citations:- source: Remote Work Policy.pdf content: Employees shall submit a remote work request at least five (5) business days prior to the requested start date.
The only response citation matches the ground-truth reference from Remote Work Policy.pdf — all expected evidence passages are cited.
The only response citation matches the ground-truth reference — there are no spurious citations.
Both context recall and precision are perfect.
Remote Work Policy.pdf, the only expected source, appears in the response citations.
The only cited document, Remote Work Policy.pdf, is among the expected sources.
Both source recall and precision are perfect.
Correct document cited, but wrong passage
| Citation | Evidence Passage |
|---|---|
| Employment Contract.pdf | Employees are required to provide a minimum of four (4) weeks written notice of resignation. |
Employment Contract.pdf
- citations:- source: Employment Contract.pdf content: Employees on probation are not entitled to severance pay upon termination within the first ninety (90) days of employment.
The cited passage is about probation and severance, not resignation notice — it does not match the ground-truth evidence passage.
The response citation's evidence passage does not match the ground-truth reference — the cited passage refers to probation and severance rather than resignation notice.
Both context recall and precision are zero.
Employment Contract.pdf, the only expected source, appears in the response citations.
Employment Contract.pdf is the expected source, so the document-level citation is correct even though the passage is wrong.
Both source recall and precision are perfect.
Mixed result - multiple documents, partial coverage
| Citation | Evidence Passage |
|---|---|
| Data Security Policy.pdf | Remote workers must use a company-approved VPN at all times when accessing internal systems from outside the office. |
| Remote Work Policy.pdf | All devices used for remote work must have full-disk encryption and approved endpoint security software installed. |
| Acceptable Use Policy.pdf | Employees must not store confidential company data on personal devices or unapproved cloud storage services. |
Data Security Policy.pdf, Remote Work Policy.pdf, Acceptable Use Policy.pdf
- citations:- source: Data Security Policy.pdf content: Remote workers must use a company-approved VPN at all times when accessing internal systems from outside the office. - source: Remote Work Policy.pdf content: Employees shall submit a remote work request at least five (5) business days prior to the requested start date. - source: IT Compliance Handbook.pdf content: Annual security training is mandatory for all employees with access to sensitive data.
Only the Data Security Policy.pdf passage matches its ground-truth reference (1 of 3). The Remote Work Policy.pdf citation uses a wrong passage, and Acceptable Use Policy.pdf is not cited at all.
Only the Data Security Policy.pdf citation matches a ground-truth reference (1 of 3). The Remote Work Policy.pdf citation uses a wrong passage, and IT Compliance Handbook.pdf has no matching ground-truth reference.
Context recall and precision are equal at 0.33.
Data Security Policy.pdf and Remote Work Policy.pdf are cited, but Acceptable Use Policy.pdf is not (2 of 3 expected sources).
Data Security Policy.pdf and Remote Work Policy.pdf are expected sources, but IT Compliance Handbook.pdf is not (2 of 3 cited sources are expected).
Source recall and precision are equal at 0.67.