Skip to content

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

Sep 2026 · arXiv.org · Vol abs/2609.34972 · 0 citations · 41 references
Computer Science

TL;DR

This work proposes $\delta$-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval, and achieves higher accuracy than visual token pruning baselines at comparable or lower computation.

Abstract

Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose $\delta$-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, $\delta$-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.

View source

Similar papers

Preprint Sep 2026

S$^2$Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

S$^2$Prune is proposed, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure and achieves the highest average accuracy among the evaluated training-free pruning methods.

Yuan-Yuan Jia, Shun-Pu Tang, Qian-Qian Yang · 1 citation · ⚡1

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

AdaVSkip is proposed, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules, and maintains strong task performance with substantially less computation.

Yu-Yao Sun, Tao Deng, Shuang-Hua Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

SinkPruner is proposed, a training-free visual token pruning framework for efficient MLLM inference that follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains toke...

Shi-Yu Li, Zi-Yuan Hu, Shijia Huang et al. · 2 citations
Preprint Sep 2026

StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models

Visual prefixes account for a major portion of the per-layer computation in multimodal large language models (MLLMs), making visual-token pruning a direct approach to accelerating inference. Existing top-K methods typically evaluate tokens independently and apply a uniform budget to all inputs, overlooking both selecti...

Han-Sen Zhang, Lan He, Min Yao et al. · 0 citations

CoViST: Visual Token Compression via Composable States

CoViST is proposed, a training-free framework that represents a compressed image as a composable visual state that retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder.

Qi Zhang, Xian-Dong Meng, Rong-Gang Wang et al. · 0 citations
#small language model Preprint Aug 2026

VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

VisLens (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head, and which matches or exceeds prior baselines while delivering a substantial latency advantage.

Jingfeng He, Sanghwan Kim, Zeynep Akata · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.