HALLMARK: Citation Hallucination Benchmark

HALLMARK is a citation hallucination benchmark for evaluating reference-verification tools on ML papers. It provides 2,525 annotated entries spanning 14 hallucination types across 3 difficulty tiers, with 6 sub-tests per entry for fine-grained diagnosis.

Difficulty Tiers

Tiers reflect how difficult a hallucination is to detect: Easy entries have obvious errors (e.g., nonexistent DOIs), Medium entries require cross-referencing metadata, and Hard entries involve subtle misattributions that demand deep semantic understanding of the cited work.

Built-in Baselines

• DOI-only — resolves DOIs and flags missing or broken links
• bibtex-updater — metadata-based verification via BibTeX normalization
• LLM-based — direct LLM prompting for citation plausibility
• Ensemble — combines DOI + metadata + LLM signals
• HaRC — hallucinated reference classification baseline
• verify-citations — end-to-end citation verification pipeline

Metrics

• Detection Rate — fraction of hallucinated citations correctly flagged
• F1 — harmonic mean of precision and recall
• Tier-weighted F1 — F1 weighted by difficulty tier (Hard counts more)
• detect@k — detection rate within the top-k ranked citations
• ECE — expected calibration error of confidence scores

Final system rankings use Plackett–Luce ranking to aggregate across metrics and tiers into a single ordering.

Design Principles

HALLMARK draws on best practices from established code and ML benchmarks: HumanEval (functional correctness via test cases), SWE-bench (real-world task grounding), LiveCodeBench (contamination-resistant temporal splits), and ONEBench (multi-metric aggregation into a unified score).