all tags

nlp-evaluation

22 notes carry this tag — 15 claims, 1 observation, 1 question, 5 captures. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

• Liang, Hu, Zhao, Song, Griffiths & Fisac (2025), 'Machine Bullshit,' is a technical ML paper that cites Frankfurt and Hicks et al. by name, but its own taxonomy does not fold in citation or reference fabrication
  claim seedling Tier 1 2026-11-05 · citation-hallucination, reference-hallucination, bullshitting, frankfurt, fabrication
• Naser (2026)'s cross-model citation-fabrication audit cites no Frankfurt, Hicks et al., or bullshitting/interpretability literature, widening the confirmed gap beyond one paper
  claim seedling Tier 1 2026-11-05 · citation-hallucination, reference-hallucination, bullshitting, frankfurt, fabrication
• Does any technical ML paper on citation/reference fabrication (beyond Rao, Wong & Callison-Burch 2026) cite Frankfurt, Hicks et al., or the interpretability 'bullshitting' literature?
  capture promoted 2026-11-04 · citation-hallucination, reference-hallucination, bullshitting, frankfurt, fabrication
• Kalai & Vempala's own proof construction, and both papers' worked examples, model arbitrary-fact hallucination as a uniform random draw over an exponentially large false-candidate space — a structural argument for divergence, not convergence
  claim seedling Tier 1 2026-11-03 · ai-safety, hallucination, fabrication, correlated-errors, calibration
• Kalai & Vempala's STOC 2024 hallucination bound is a rate theorem sorted by a fact's statistical regularity, not a theory of task topic or error regime
  claim seedling Tier 1 2026-11-03 · ai-safety, hallucination, fabrication, calibration, nlp-evaluation
• Kalai & Vempala's theorem is silent on cross-model hallucination agreement, so Correlated Errors' convergent-fabrication finding is neither modeled nor anticipated by it
  claim seedling Tier 1 2026-11-03 · ai-safety, hallucination, correlated-errors, algorithmic-monoculture, calibration
• Does Kalai & Vempala's 2023/STOC-2024 hallucination bound predict which regime — convergent or divergent fabrication — a given task should produce?
  capture promoted 2026-11-02 · ai-safety, hallucination, fabrication, correlated-errors, algorithmic-monoculture
• Does AI-safety/interpretability literature have a specific name and mechanism for plausible-sounding fabricated intermediate claims — and does it explain why two independent generations converge on the same one?
  capture promoted 2026-10-31 · ai-safety, interpretability, hallucination, fabrication, correlated-errors
• LLM errors are correlated well above chance, and the correlation rises, not falls, as paired models get individually more accurate
  claim seedling Tier 1 2026-10-31 · ai-safety, correlated-errors, algorithmic-monoculture, llm-evaluation, nlp-evaluation
• Higher 'generative-process diversity' between a model pair predicts lower correlated failure, beyond semantic similarity or capability — a single, not-yet-corroborated 2026 preprint
  claim seedling Tier 1 2026-10-31 · ai-safety, correlated-errors, algorithmic-monoculture, representational-similarity, nlp-evaluation
• Divergence, not convergence, is the literature's own default outcome for an open-ended fabricated detail with no strong training-data regularity behind it
  claim seedling Tier 1 2026-10-31 · ai-safety, hallucination, fabrication, correlated-errors, nlp-evaluation
• Citation-specific 'fabricated snippets' and reasoning-specific 'bullshitting' remain separately named, unconnected mechanisms in the literature
  observation seedling Tier 1 2026-10-31 · ai-safety, interpretability, hallucination, fabrication, citation-hallucination
• Does an independent source corroborate RIKER's (Roig, 2026) 32K-context hallucination-rate range?
  capture promoted 2026-10-22 · rag, benchmark, nlp-evaluation, hallucination, source-corroboration
• RIKER's widely-cited '7.9% to 96.3%' hallucination-rate range is a 128K-context figure, not 32K — the paper's own 32K range is 0.0%-93.2%
  claim seedling Tier 1 2026-10-22 · rag, benchmark, nlp-evaluation, hallucination, quote-provenance
• No independent source corroborates RIKER (Roig, 2026) — the only follow-up work is same-author/same-company, and the author's own later paper says independent replication has not happened
  claim seedling Tier 1 2026-10-22 · rag, benchmark, nlp-evaluation, hallucination, source-corroboration
• RIKER author's own larger self-replication does not reproduce the original GLM-4.5-vs-Qwen3-Next-80B fabrication gap — both models land in the same low-fabrication tier at 32K
  claim seedling Tier 1 2026-10-22 · rag, benchmark, nlp-evaluation, hallucination, self-replication
• Has AI-safety/NLP-evaluation literature named and studied tool-access fabrication as distinct from ordinary hallucination?
  capture promoted 2026-10-21 · ai-safety, nlp-evaluation, hallucination, tool-use, fabrication
• Do Rosario, Younes & Pires (2026) define 'grounding violation' as distinct from 'factual hallucination,' and name 'parametric leakage' as the mechanism that turns an ungrounded answer into confident fabrication
  claim seedling Tier 1 2026-10-21 · rag, nlp-evaluation, hallucination, grounding, ai-safety
• Rao, Wong & Callison-Burch (2026) name 'fabricated snippets' as a citation-reliability failure mode distinct from URL hallucination, and flag it in their own words as not yet systematically studied
  claim seedling Tier 1 2026-10-21 · citation-hallucination, reference-hallucination, nlp-evaluation, source-reliability, ai-safety
• RIKER (Roig, 2026) finds grounding accuracy and fabrication resistance are empirically separable capabilities, not two faces of one hallucination score [unverified-quant]
  claim seedling Tier 1 2026-10-21 · rag, benchmark, nlp-evaluation, hallucination, grounding
• Sethi et al. (2026) name 'fabrication after tool failure' and find its trigger is structural (whether the tool's response signals failure), not lexical, topical, or instructed
  claim seedling Tier 1 2026-10-21 · ai-safety, tool-use, hallucination, fabrication, benchmark
• Does an independent source corroborate RIKER's (Roig, 2026) 32K-context hallucination-rate range (7.9%-96.3% across models, GLM-4.5 0.0% vs Qwen3-Next-80B 40.9% fabrication at comparable extraction accuracy)?
  question answered 2026-10-21 · rag, benchmark, nlp-evaluation, hallucination, source-corroboration