Mitigating Hallucinations in Large Multimodal Reasoning Models by Regulating Parametric Prior Drift
Abstract
Large Multimodal Reasoning Models (LMRMs) have recently shown strong capabilities in complex visual understanding and multi-step inference. However, they remain highly vulnerable to multimodal hallucinations over extended reasoning trajectories. In this work, we study this failure as a state-dynamics problem: as reasoning unfolds, hidden states become progressively less grounded in visual evidence and increasingly dominated by parametric priors retrieved through Feed-Forward Networks (FFNs). Our analysis reveals that this visually ungrounded drift accumulates over long reasoning chains and shows consistent association with FFN residual updates. Based on this view, we propose Adaptive Fine-grained Residual Gating (AFRG), a training-free controller that regulates harmful FFN updates during inference. Specifically, AFRG measures how each FFN residual update changes alignment with the most relevant visual tokens and uses an adaptive bounded gating function to suppress updates that pull hidden states away from the image-supported evidence. Across four hallucination benchmarks and three general multimodal benchmarks on Qwen3-VL and InternVL3.5, AFRG reduces MMHal hallucination rate from 0.34 to 0.26, improves CHAIR_I by 3.8 points, and raises MME Total by 38.3 points while preserving general multimodal capability. These results highlight FFN-level prior regulation as a promising inference-time direction for faithful multimodal reasoning.