WM-VLM is introduced, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states and suggests that internal world models offer a promising path toward VLMs that reason in both language and visual space.
Abstract
Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional vision-language models (VLMs) reason primarily through language. We investigate whether VLMs can solve spatial problems by reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for reasoning. We programmatically construct spatial reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both language and visual space. Code is publicly available at: https://github.com/yuh-zha/WM-VLM
It is found that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both, and gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging.
Lars Benedikt Kaesberg, Tianyu Yang, F. Wunderlich et al.· 1 citation
A small detector is trained to predict the internal development of answer readiness and stop reasoning at that point, connecting the internal development of answer readiness to a practical rule for allocating reasoning computation.
This work uses the Qwen3-VL-4B (Bai et al., 2025), a modern VLM, to decode how visual information is encoded across depths and proposes a synthetic dataset of simple geometric shapes for controlled analysis, along with queries crafted to precisely test language cues.
Adhithya Laxman Ravi Shankar Geetha, Aulia Kharis Rakhmasari, Haleema Ramzan et al.· 0 citations
Locator-Critic (LOCI) is proposed, a training-free framework that decouples visual search from evidence verification and improves accuracy for both open-weight models like Qwen3-VL and proprietary models like Gemini 2.5 Pro.
Walid Bousselham, Mathilde Caron, Arsha Nagrani et al.· 0 citations
Across six diverse visual reasoning tasks including multi-view spatial reasoning and collision prediction, ReImaGin consistently outperforms both text-only reasoning and specialist vision-tool baselines, with gains of up to 25\%, demonstrating the advantage of flexible, generative visual reasoning.
Nishad Singhi, Hector P. Rodriguez, Aditya Arora et al.· arXiv.org· 0 citations
Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model, is proposed, endowing models with native fine-grained region description and flexible reasoning capabilities.
Chang-Jiang Jiang, Qian-Nian Zhao, Lei Xin et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.