Visual Attention Faithfulness in Vision-Language Models is Heterogeneous
Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and suffic...
Xu-Rui Song, Wei-Shi Wang, Zhong-Qi Yue et al.
· 2 citations