Skip to content
Book Open access

Attention Switch is All You Need: Collaborative Token and KV Compression for Long Video Inference in Hybrid MLLMs

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 31 references

Abstract

Multimodal Large Language Models (MLLMs) often adopt a hybrid Mamba-Transformer architecture to balance accuracy and inference efficiency. However, when processing long-video inputs, the interdependence between Mamba and Transformer layers renders traditional optimization techniques ineffective, as they are designed exclusively for single-architecture models. In this paper, we observe that Mamba and Transformer exhibit both consistent and complementary behaviors during inference. Building on these novel observations, we propose DeltaInfer, an attention-switch ?-guided collaborative compression framework for efficient inference. DeltaInfer operates in two stages. It first performs frame-level semantic condensation in Mamba layers to prune redundant backgrounds. It then selectively skips token-level Key-Value (KV) generation in Transformer layers for tokens whose semantics have been already encoded in Mamba's hidden states. This design jointly reduces prefill FLOPs and KV cache memory footprints. Both stages measure importance using the built-in parameter ?, which is natively available in Mamba layers, and therefore require neither auxiliary model inference nor attention-matrix materialization, unlike conventional attention-guided approaches. The importance threshold as a hyperparameter is adaptively determined in a data-driven manner, to ensure generalization. Extensive experiments on three hybrid MLLMs across five video understanding benchmarks demonstrate that DeltaInfer achieves 1.92× end-to-end acceleration and a 63.5% reduction in peak GPU memory, while reducing model-level average accuracy by less than 0.15 percentage points. DeltaInfer establishes a plug-and-play paradigm for efficient long-video inference in hybrid MLLMs.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.