Skip to content
Preprint

Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere

Aug 2026 · 0 citations · 32 references
Computer Science

TL;DR

This work proposes a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs.

Abstract

Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input. These tokens often contain substantial redundancy arising from repeated visual patterns, leading to unnecessary computation in the subsequent language-model processing. Existing token compression methods, including pruning and merging, perform compression online during inference, repeatedly incurring additional computation for each input video and often relying on model-specific designs that limit their generality, we instead rethink this paradigm by shifting the costly compression process offline. We propose \textbf{ONCE}, a plug-in video token compression framework that introduces an offline-to-online paradigm: a frequency-aware global codebook is learned once in the visual feature space and reused for lightweight online compression through codebook lookup and aggregation, reducing repeated per-video computation and the need for model-specific compression designs. Extensive experiments across multiple video understanding benchmarks and against diverse compression baselines demonstrate that our approach achieves a strong accuracy-efficiency trade-off, maintaining competitive performance while achieving the lowest inference latency among compared methods.

View source

Similar papers

Preprint Sep 2026

Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline

Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruni...

Xiao Zhang, Wang Zeng, Sheng Jin et al. · 0 citations
Preprint Sep 2026

Video Compression with Graph-inspired Neural Representation

The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86\% and 14.68\% (in BD-rate), respectively, measured by PSNR on the UVG dataset.

Chang-Qiu Wang, Ge Gao, Fan Zhang et al. · 0 citations
Preprint Aug 2026

Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure, and adapts this construction along task and spatial axes to preserve visual information dispersed across frames under compress...

Wen-Ti Yin, Xiao-Tian Han, Jun-Yuan Shang et al. · 0 citations

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mism...

Yulong Liu, Xiao-Tian Han, Jun-Yuan Shang et al. · 0 citations
Preprint Aug 2026

Visual Token Coding for Video Multimodal Large Language Models

In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals...

Chenxin Fang, Tao Chen, Junshuang You et al. · 0 citations

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

KeyRec is a training-free framework for constructing bounded visual memory that preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank and achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual...

Zi-Han Chen, Xuejian Rong, Xiao-Juan Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.