Skip to content
Preprint

CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

Oct 2026 · 0 citations · 25 references
Computer Science

TL;DR

CatchUp is proposed, a training-free framework combining reversible visual-token routing with Local Transition Correction that achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets.

Abstract

Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing visual-token reduction methods often irreversibly discard tokens despite depth-varying query relevance. Although recoverable routing allows excluded tokens to return, re-admitted tokens retain stale hidden states after bypassing intermediate decoder transformations. Thus, token recoverability does not guarantee representation readiness at the current decoder depth. Under a fixed routing trajectory at a 64-token budget, replacing stale re-admitted states with current-depth oracle representations improves average RefCOCO Acc@0.5 from 18.96 to 23.18, revealing the impact of representation staleness. We propose CatchUp, a training-free framework combining reversible visual-token routing with Local Transition Correction (LTC). CatchUp repeatedly re-evaluates query-conditioned relevance over the complete visual-token set, preserving deferred tokens for later re-admission. LTC approximates their skipped representation evolution by transferring reference-to-current hidden-state transitions from semantically and spatially related active anchors. This correction improves alignment with current-depth oracle representations without recomputing the skipped decoder layers. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets. At a 64-token budget, CatchUp processes 88.9% fewer active visual tokens per post-routing decoder interval, retains 92.17% of dense-model multimodal performance, and reduces end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp

View source

Similar papers

Book Open access Nov 2026

DATP: Training-Free Depth-Aware Token Pruning for Vision-Language Models

The massive redundancy of visual tokens in Vision-Language Models (VLMs) creates a critical computational bottleneck during large language model (LLM) prefill. Recent training-free pruning methods provide plug-and-play acceleration, yet their pruning decisions often rely on limited visual states or delayed in-LLM signa...

Qing-Xin Liu, Jin-Zhi Liao, Xiang Zhao · 0 citations
Sep 2026

Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models

This work proposes Feature-Aware Token Attack (FATA), which couples attention suppression with cosine-based feature preservation on a fixed set of salient clean-image tokens and has the lowest measured detection rate among four attacks across three evaluated detectors at a 5% false-positive rate.

Shilinlu Yan, Bo-Wen Chen, Yuechen Zhang et al. · 0 citations
#small language model Preprint Sep 2026

SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

Reasoning vision-language models (VLMs) process images and videos as long sequences of visual tokens, making inference expensive. Training-free token pruning reduces this cost, but aggressive compression can sharply degrade performance, often attributed to irreversible loss of task-relevant visual information. We show...

Ahmadreza Jeddi, En-Ming Zhang, Jasper Gerigk et al. · 0 citations

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression, demonstrating its practicality for resource-constrained LVLM deployment.

Guang-Chuan Lv, Dian-Xing Shi, Ding-Jie Fu · 0 citations

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

KeyRec is a training-free framework for constructing bounded visual memory that preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank and achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual...

Zi-Han Chen, Xuejian Rong, Xiao-Juan Wang et al. · 0 citations
Preprint Oct 2026

Query-adaptive Recurrent Tokenization

Visual information worth extracting from an image and the token budget it warrants depend on the text query. Yet modern vision and vision-language models largely encode images independently of the query and allocate a fixed visual token budget to every input. We introduce the Query-Adaptive Recurrent Tokenizer (QART),...

Bavesh Balaji, P. Gopinath, Pouya Sadeghi et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.