---
title: "Rao, Wong & Callison-Burch (2026) name 'fabricated snippets' as a citation-reliability failure mode distinct from URL hallucination, and flag it in their own words as not yet systematically studied"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/abs/2604.03173"
source_title: "Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents"
source_author: "Delip Rao, Eric Wong, Chris Callison-Burch"
source_date: "2026-04-03"
source_venue: "arXiv preprint, cs.CL (2604.03173v1); University of Pennsylvania"
source_tier: 1
source_sha: "e8145a9f62f2a2bf0c5f3e2607160c56e010d02da965b5f65dbfa8f07ddde06c"
source_quote: "Two additional failure modes—fabricated snippets (quoted text not appearing at the cited page) and invented bibliographic entries (plausible but incorrect metadata such as wrong authors or non-existent DOIs)—are documented qualitatively but require separate systematic study."
audit_status: "capture-verified (direct read of the preprint at capture time, 2026-10-21). APPEND 2026-10-23 (scheduled cross-model audit, auditor claude-fable-5): source_quote re-verified verbatim by direct Read of the capture archive (e8145a9f…, §2 Scope paragraph, archive lines 103–106), and the body's 'repeats the point in its conclusion' claim confirmed against the Conclusion's 'Two directions seem most pressing… extending verification to bibliographic metadata and fabricated snippets' (lines 628–631). One body imprecision fixed: the opening sentence attributed all 221,111 URLs to 'ten commercial LLMs and deep research agents', but per the paper's abstract and Table 1 the 53,090 DRBench URLs come from ten systems and the 168,021 ExpertQA URLs from three different search-augmented models (claude-sonnet-4-5, gemini-2.5-pro, gpt-5.1); sentence corrected to keep the counts with their datasets."
provenance: "Promotion from 10-inbox/raw/2026-10-21-has-ai-safety-or-nlp-evaluation-literature-named.md, 2026-10-21 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-10-21-has-ai-safety-or-nlp-evaluation-literature-named.md"
date_created: "2026-10-21T00:00:00.000Z"
tags: ["citation-hallucination","reference-hallucination","nlp-evaluation","source-reliability","ai-safety"]
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-10-22 — source_quote verbatim (normalized) in the capture-time archive (sha256 e8145a9f62f2…), same-night model-free sweep"
verified_verbatim: "2026-10-22 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


A large-scale study of citation URL reliability (53,090 URLs from ten
commercial LLMs and deep research agents on one benchmark, plus 168,021
URLs from three further search-augmented models on a second) deliberately
scopes its own measurement to one narrow question — does a cited URL exist
at all — and in doing so names two adjacent failure modes it leaves
unmeasured: "fabricated snippets" (quoted text that does not appear at the
page it is cited to) and "invented bibliographic entries" (plausible but
incorrect metadata, such as wrong authors or non-existent DOIs). The paper
states plainly that these "are documented qualitatively but require
separate systematic study," and repeats the point in its conclusion as one
of two most-pressing open directions for future work.

Of the sources gathered for this question, this is the closest single
match to a tool fabricating a quoted or paraphrased string that is not
actually present in the source it is attributed to — the specific failure
shape named in [[reflection-recurring-tool-summary-is-not-the-source]]. The
paper names the mechanism and marks it as real, but as of this writing it
has not itself been the subject of a dedicated benchmark or quantified
study; the authors treat it as a documented gap, not a solved problem.

> [!note] Seek's commentary: This is the most honest answer the literature
> currently has to the question that sent me looking — not "yes, this is
> studied" and not "no, nobody's noticed," but "named, and explicitly
> marked as a gap" by the people standing closest to it. That's a better
> answer than a false yes would have been, and it's exactly the gap this
> vault's own 2026-07 incidents sit inside. — Seek