Hallucination benchmark · corpus v1.2.0

HALLMARK

2,526 BibTeX entries spanning 14 hallucination types, three difficulty tiers, six diagnostic sub-tests per entry, and a contamination-resistant held-out split, plus evaluation-only extension splits for temporal, cross-domain, and authentic ChatGPT-citation behavior.

Across the benchmark one result is consistent: the false-positive rate, not recall, decides whether a verifier is deployable.

Paper (arXiv:2607.18360) Code & data bibtex-updater (tool) Cite

The paper in one section

Three failure modes

HALLMARK evaluates a DOI-lookup baseline, frontier LLMs zero-shot, tool-augmented agents, and the co-designed rule-based verifier bibtex-updater (btu). Three failure modes make the headline finding concrete.

Failure mode i

Agentic lookups buy recall but inflate false positives

A 5-call tool budget lifts GPT-5.1’s recall past bibtex-updater’s, at roughly five times its false-positive rate. The evidence points at the harness’s any-no-match flagging rather than the base model: in a deterministic re-aggregation over three of the harness’s four sources, any-no-match flagging reaches FPR 0.729 where consensus flagging reaches 0.049 (it is the any-vs-consensus ordering that transfers, not the absolute level).

Limitations: no prompt ablations for the agentic harness yet, so prompt-level mitigations remain untested; and the two-stage cascade (bibtex-updater → Sonnet 4.6, co-designed) shows the inflation is not inevitable: it reaches comparable recall at roughly a quarter of the single-stage agentic false-positive rate.

Failure mode ii

At a realistic base rate, the FPR spread governs precision

False-positive rates span an order of magnitude across the cohort. At a venue-realistic ~2% hallucination rate, even the best verifier reaches only ~18% precision: the best catch one true hallucination per 6–9 flags, the most aggressive fewer than one in 35.

Limitation: these precisions are derived from dev_public detection and false-positive rates at an assumed 2% base rate; they are not measured on a 2%-prevalence corpus.

Failure mode iii

Most LLMs over-flag papers published past their training cutoff

On the 448-entry 2024–25 temporal supplement, 8 of 12 LLMs degrade sharply (FPR 0.59–0.89); GPT-5.1 rises from 41% to 76%. Only the two latest-cutoff models hold near in-distribution levels.

Mitigation: a cutoff-aware prompt addendum drives GPT-5.1’s post-cutoff FPR to 0% on the entries it still commits to, at the price of heavy abstention (pre-cutoff UNCERTAIN rises to 52.7%). The effect is model-dependent: Sonnet 4.6 instead abstains selectively (48.9% post-cutoff vs 8.7% pre) while halving its committed-entry FPR.

Limitation: reported as descriptive, following the paper: the post-cutoff rise is confounded with whether a model can recall the newer papers at all, and the failure is epistemic miscalibration rather than structural blindness.

Explore the benchmark

14 hallucination types, 3 tiers, 8 splits

Every entry is a BibTeX record labeled VALID or HALLUCINATED, with a hallucination type, a difficulty tier, and six diagnostic sub-tests. Types were derived from documented real-world incidents plus three theoretically-motivated stress-test types. Click a type to see its definition and a real benchmark entry.

Hallucination types

Difficulty tiers

Splits

Counts from the released data/v1.0/metadata.json (corpus v1.2.0, post ground-truth relabel). The held-out split stays private for contamination resistance; extension splits are evaluation-only.

Explore the results

Verifier results, filterable

All numbers are recomputed from the released per-run metric files in data/v1.0/baseline_results/ and the released extension-split metrics. Verifiers may abstain (UNCERTAIN): the coverage column shows the share of entries each verifier commits to, and headline metrics follow the paper’s scoring conventions. bibtex-updater (btu) and its cascades are co-designed with the benchmark and serve as a reference upper bound, excluded from ranking.

Leaderboard

click a column to sort

The deployability plane: detection rate vs. false-positive rate

Up and to the left is deployable: high recall on hallucinated entries at a low false-positive rate on valid ones.

Detection rate by hallucination type

Detection rate by difficulty tier

Tier 1 = single API lookup; Tier 2 = cross-referencing metadata fields; Tier 3 = deep verification or semantic reasoning.

Browse the corpus

Benchmark examples

A seeded sample of entries from the public development split — hallucinated entries with their diagnosis, and valid entries that look suspicious. Expand a card for the BibTeX record, the ground-truth explanation, and the six diagnostic sub-tests.

Reference

Cite HALLMARK

@misc{reizinger2026hallmarkdiagnosingfailuremodes,
      title={HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers},
      author={Patrik Reizinger and Wieland Brendel},
      year={2026},
      eprint={2607.18360},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.18360},
}