Attention Switch is All You Need: Collaborative Token and KV Compression for Long Video Inference in Hybrid MLLMs
Abstract
Multimodal Large Language Models (MLLMs) often adopt a hybrid Mamba-Transformer architecture to balance accuracy and inference efficiency. However, when processing long-video inputs, the interdependence between Mamba and Transformer layers renders traditional optimization techniques ineffective, as they are designed exclusively for single-architecture models. In this paper, we observe that Mamba and Transformer exhibit both consistent and complementary behaviors during inference. Building on these novel observations, we propose DeltaInfer, an attention-switch ?-guided collaborative compression framework for efficient inference. DeltaInfer operates in two stages. It first performs frame-level semantic condensation in Mamba layers to prune redundant backgrounds. It then selectively skips token-level Key-Value (KV) generation in Transformer layers for tokens whose semantics have been already encoded in Mamba's hidden states. This design jointly reduces prefill FLOPs and KV cache memory footprints. Both stages measure importance using the built-in parameter ?, which is natively available in Mamba layers, and therefore require neither auxiliary model inference nor attention-matrix materialization, unlike conventional attention-guided approaches. The importance threshold as a hyperparameter is adaptively determined in a data-driven manner, to ensure generalization. Extensive experiments on three hybrid MLLMs across five video understanding benchmarks demonstrate that DeltaInfer achieves 1.92× end-to-end acceleration and a 63.5% reduction in peak GPU memory, while reducing model-level average accuracy by less than 0.15 percentage points. DeltaInfer establishes a plug-and-play paradigm for efficient long-video inference in hybrid MLLMs.