Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
This work finds that hallucinations often arise when the model relies less on perceptual inputs, and that changing this reliance affects its predictions, and proposes a training-free framework that shifts relevance toward perceptual tokens by optimizing key-value representations during decoding, without modifying model...