Skip to content

Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

Sep 2026 · arXiv.org · Vol abs/2609.28991 · 0 citations · 38 references
Computer Science

TL;DR

A causal stage-intervention protocol that overwrites individual stages while holding the downstream task fixed is introduced, finding that grounding is the dominant source of downstream error, with roughly four times the causal impact of corrupting visual observations.

Abstract

Video understanding is increasingly performed by multi-stage LLM agents that separate temporal grounding, visual observation, and reasoning. Yet these stages are typically evaluated on different benchmarks and distributions, making it difficult to determine where hallucinations originate. We first organize existing benchmarks around these stages and show that their scores provide inconsistent diagnostic signals: stronger stage-level performance does not reliably imply lower downstream hallucination, and even benchmarks targeting the same capability can disagree. We therefore introduce a causal stage-intervention protocol that overwrites individual stages while holding the downstream task fixed. Across 60,008 runs on three video-agent architectures, we find that grounding is the dominant source of downstream error, with roughly four times the causal impact of corrupting visual observations. Successful grounding depends primarily on locating the correct region rather than precise temporal overlap, explaining why standard mIoU metrics poorly predict downstream reliability. We further find that incorrect evidence is substantially more harmful than missing evidence. Finally, auditing existing benchmarks against these interventions reveals that their scores do not reliably predict causal cascade sensitivity and can fail under distribution shift. These results motivate intervention-based, stage-aware evaluation for trustworthy video agents.

View source

Similar papers

Preprint Sep 2026

What Do Hallucinations Reveal About Multimodal Reasoning? Diagnosing Visual Grounding Failures via Contrastive Decoding Probes

When strong multimodal models are widely available, progress requires new scientific methodologies beyond benchmark scores---using models as instruments for understanding behavior. We address this by asking: can we use large vision-language models (LVLMs) as experimental instruments for studying their own failure dynam...

Zhi-Peng Zhao, Wen-Xu Wang, Peishun Liu et al. · 0 citations
#artificial intelligence Preprint Aug 2026

EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation

EviAnchor is proposed, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation in large vision-language models and demonstrates consistent improvements in visual grounding.

Sihang Jia, Shu-Liang Liu, Song-Bo Yang et al. · 0 citations
#artificial intelligence Preprint Aug 2026

ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models

ReWEIGH is a training-free decoding intervention that aggregates vocabulary ranks across visual positions and compares each candidate with a token-specific reference estimated from unlabeled images and applies a bounded penalty only to candidates that fall below their reference.

Jihae Jeong, Jun-Ha Choi, Hwanjo Yu · 0 citations
#artificial intelligence Preprint Oct 2026

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve...

Qi-Ji Zhou, Jun-Yi Hu, Pei-Yu Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions requir...

Hyunjae Ra, Aecheon Jung, Jungin Park et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.