Skip to content

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

Sep 2026 · arXiv.org · Vol abs/2609.30783 · 0 citations · 30 references
Computer Science

TL;DR

LIRSeg is proposed, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation, and introduces three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates for learning complementary representations, and latent diversity amplification for preventing representational collapse.

Abstract

Reasoning segmentation aims to interpret implicit textual queries and enable fine-grained visual perception, which is critical for applications such as human-computer interaction and embodied agents. Existing methods typically generate explicit Chain-of-Thought (CoT) by multimodal large language models (MLLMs) before localizing the target. Although intuitive, such explicit verbal reasoning introduces substantial attention interference: redundant textual tokens disrupt attention during perception-token generation and also increase the effective distance between visual tokens. To address this issue, we propose LIRSeg, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation. LIRSeg is trained in two stages: spatial alignment grounds the latent tokens in object-relevant visual evidence, and GRPO further optimizes them with segmentation rewards. To make these compact latent tokens more informative, we introduce three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates for learning complementary representations, and latent diversity amplification for preventing representational collapse. Extensive experiments on benchmarks demonstrate that LIRSeg consistently improves both segmentation accuracy and reasoning efficiency. Compared with the VisionReasoner baseline, LIRSeg achieves absolute gIoU improvements of 4.9% on ReasonSeg, 7.1% on MUSE, and 4.7% on MMR, while achieving a approximately 16x reduction in reasoning tokens. Code is available in supplementary materials.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows

Multimodal large language models (MLLMs) often answer visual reasoning questions by relying on linguistic priors rather than task-relevant visual evidence. Textual chain-of-thought reasoning can partially mitigate this issue by encouraging models to decompose visual questions into intermediate evidence-seeking steps, b...

Albert Gao, Bing Xue, Andrea Zanette · 1 citation
Preprint Aug 2026

GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning

GLaQ, a grounded latent-query framework that replaces sequential latent rollout with a fixed set of context-conditioned queries grounded in the original visual tokens, is introduced, suggesting that direct query-to-image grounding can recover localized evidence from the full image without external visual operations or...

Ze-Sheng Yang, Lingling Zhang, Xinyu Zhang et al. · 1 citation
Sep 2026

Distilling Visual Reasoning into Text Space

V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning.

Wen-Han Yang, Nilay Naharas, Ali Payani et al. · 0 citations

TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning

TReVS is proposed, a training-free framework that combines textual relevance with vision-encoder saliency for pre-LLM pruning and leverages high-variance attention heads to remove task-irrelevant tokens at shallow-to-intermediate layers of the LLM.

Jing Wang, Zhi-Ping Wu, Dong-Dong Ren et al. · 0 citations
Preprint Oct 2026

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as...

Yi-Cheng Xue, Han Wu, Ju-Feng Yang et al. · 1 citation

Question-Specific Knowledge Graphs for Efficient Visual Reasoning

Recent work in visual question answering has shown that vision-language models can exhibit strong reasoning capabilities by translating visual inputs into textual representations. The effectiveness of this translation depends on how well visual details are retained; models need to surface and align both explicit and im...

Ting-Chih Chen, Emile van Krieken, Shu-Jian Yu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.