Skip to content

Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning

Sep 2026 · arXiv.org · Vol abs/2609.28851 · 0 citations · 28 references
Computer Science

TL;DR

FlipDir (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding and consistently outperforms existing methods on the combined recovery and preservation metric.

Abstract

Vision-language models (VLMs) achieve strong visual reasoning performance, yet subtle changes from routine image capture and processing can alter their reasoning trajectories even when images appear nearly identical. In long-horizon generation, the resulting activation shifts may accumulate across decoding steps, progressively altering reasoning tokens and ultimately changing the final answer, a phenomenon referred to as answer flips. To address this instability, we propose FlipDir (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding. A margin-based gate limits subspace attenuation to uncertain decoding steps, recovering original predictions while preserving stable ones. To evaluate robustness beyond accuracy or consistency on fixed test sets, we introduce VisFlip, a benchmark framework that constructs evaluation groups for a target model and visual variation setting to separately assess recovery of original predictions and preservation of stable ones. VisFlip spans nine dataset-variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, covering subtle visual variations common in each domain. Experiments across 18 settings demonstrate that FlipDir consistently outperforms existing methods on the combined recovery and preservation metric. We will make our code publicly available.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

FLIP is a validation step for intervention-based mechanistic interpretability, not a steering method, for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation.

Drandreb Earl O. Juanico, Rowel O. Atienza · 0 citations

Reasoning Instructions Can Break Answer Decoding in Vision-Language Models

Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits before the model generates any rationale. We call this CoT-prefix scoring. On ScienceQA, Qwen2.5-VL-7B drops from 80.76% to 45.48%, and across five option-content permutatio...

Zeyan Li, Si-Yuan Qiu, Jian-Feng Xu · 0 citations
#natural language process... Preprint Oct 2026

Correctness Is a Direction: Geometric Answer Selection in Language Models

Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models. We show that the mean displacement from incorrect to correct answer representations, computed at approximately 70\% of model depth from fifty labeled examples with no parameter updates, yields a scoring direction...

M. Armstrong, Navid Ayoobi, Pradham Mummaleti et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs

Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much. We introduce OR-Bench, a fine-grained benchmark for object-rotation reasoning with eight tasks covering rotation detection, rotation magnitude estimation, and multi-view rotation reasoning. Across 12...

Zhao-Chen Wang, Yu-Jun Cai, Huang-Bo Zou et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.