Skip to content

Beyond Selection: Token Parameterization for Extreme Visual Token Compression

Sep 2026 · arXiv.org · Vol abs/2609.35232 · 0 citations · 46 references
Computer Science

TL;DR

Braco is a lightweight four-step coder that combines transform-basis truncation, input-independent basis-coordinate embeddings, budget-dependent orthogonal re-parameterization, and learned spatial residual tokens from lightweight pooling that forms the favorable empirical accuracy-efficiency frontier under compression.

Abstract

Visual-token compression is effective for improving the efficiency of vision-language models, but under extreme compression budgets, token pruning can break visual grounding while learned resamplers increase parameter count, attention cost, and training complexity. We revisit compression through a token parameterization lens, separating (i) basis transformation and structured truncation (retained subspace/compressibility) from (ii) coordinate organization (optimization and cross-modal alignment). This view yields two coupled objectives, compressibility and learnability, which we formalize as unified functionals. Guided by these objectives, we design Braco, a lightweight four-step coder that combines transform-basis truncation, input-independent basis-coordinate embeddings, budget-dependent orthogonal re-parameterization, and learned spatial residual tokens from lightweight pooling. Experiments show that Braco forms the favorable empirical accuracy-efficiency frontier under $23\times$--$64\times$ compression and remains competitive at $144\times$, reaching 95.2% accuracy while reducing prefill FLOPs by 84.2%--86.7% relative to the uncompressed upper bound. Against prior methods, Braco matches or improves accuracy while achieving up to approximately 36% end-to-end speedup and using $16.6\times$/$78.8\times$ lower compressor latency/FLOPs.

View source

Similar papers

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression, demonstrating its practicality for resource-constrained LVLM deployment.

Guang-Chuan Lv, Dian-Xing Shi, Ding-Jie Fu · 0 citations
#artificial intelligence Preprint Oct 2026

VETO: Video Efficient Token Optimization for Vision Language Models

The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure.

Gueter Josmy Faure, Hao-Ping Wang, Min-Hung Chen et al. · 0 citations
#machine learning Conference Open access Oct 2026

V-CoLA: Vision Token Compression with Linear Attention

Vision-language models (VLMs) have demonstrated impressive capabilities but suffer from substantial computational overhead, as vision tokens dominate the input sequence. This motivates vision token compression as a key direction to alleviate the burden. However, with the emergence of hybrid architectures incorporating...

Hao Jiang, Yi-Ru Mao, Tian-Peng Bu et al. · 0 citations
Preprint Nov 2026

SVD-Prune: Spectral Vision Token Pruning under Extreme Compression

Vision-Language Models (VLMs) and Vision-Language-Action (VLA) models process hundreds of visual tokens, making visual representations a major computational bottleneck. We argue that visual token pruning should be formulated not as identifying individually important tokens, but as preserving the dominant low-rank spect...

Yvon Apedo, Martyna Poreba, M. Szczepanski et al. · 0 citations
Preprint Oct 2026

Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs

Foveated Compression is introduced, which encodes a full-resolution image once and represents it with a mixture of native- and compressed-resolution visual tokens, and shows complementary bottlenecks in region selection and compressed-region fidelity.

Donghyun Han, Jangho Park, Yuseok Bae · 0 citations
#machine learning Preprint Oct 2026

Rethinking Sequence Compression with Test-time Regression

TTR-Compressor is evaluated as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models and improves RULER performance over attention-based compressors by a large margin.

Sihan Yang, Jie Huang, Han-Xuan Li et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.