It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all is introduced.