Skip to content

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Sep 2026 · arXiv.org · Vol abs/2609.34809 · 0 citations · 38 references
Computer Science

TL;DR

A small detector is trained to predict the internal development of answer readiness and stop reasoning at that point, connecting the internal development of answer readiness to a practical rule for allocating reasoning computation.

Abstract

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

View source

Similar papers

MM-OPD: Towards One More Bottleneck Between Perception and Reasoning

MM-OPD is introduced, a multimodal on-policy self-distillation framework for symbolic-to-visual correction that transfers guidance from symbolic-conditioned behavior to the image-conditioned policy through residual token-level targets, steering the model toward correct visual evidence.

Jin-Tao Tong, Yujing Lou, Zhan-Ming Shen et al. · 0 citations
Preprint Oct 2026

Flow-of-Thought: A Framework for Visual Reasoning

Mental imagery, ``seeing with the mind's eye''is an essential aspect of human cognition. Despite rapid progress Large Language Models (LLMs) and Vision Transformers (ViTs) still underperform on tasks requiring spatial understanding. To address this, we introduce Flow-of-Thought (FoT), a framework that integrates the ge...

Mariia Baidachna, N. Pugeault · 0 citations
Sep 2026

WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning

WM-VLM is introduced, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states and suggests that internal world models offer a promising path toward VLMs that reason in both language and visual space.

Yuheng Zha, Yi-Lei Wang, Qiyue Gao et al. · 0 citations
Preprint Oct 2026

Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm

When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to respond not say another. This becomes an issue when the image contains multiple features that equally relate to the question at hand: how would...

Nurbüke Teker, Rui Xiao, Zeynep Akata et al. · 0 citations

Reasoning Instructions Can Break Answer Decoding in Vision-Language Models

Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits before the model generates any rationale. We call this CoT-prefix scoring. On ScienceQA, Qwen2.5-VL-7B drops from 80.76% to 45.48%, and across five option-content permutatio...

Zeyan Li, Si-Yuan Qiu, Jian-Feng Xu · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.