VertiKV: Vertical-Integrity KV Cache Compression for Efficient Multimodal Long-Context Inference
Abstract
As multimodal large language models (MLLMs) are increasingly applied in scenarios with long textual contexts and high-resolution images, KV cache memory footprint during inference has grown substantially. To optimize inference efficiency, existing compression methods mostly rely on scoring-based strategies, which estimate token importance from localized attention weights and perform retention independently at each head. As a result, the distribution of retained tokens may exhibit irregular sparsity across layers. Drawing inspiration from structured pruning in model compression, we investigate how unstructured token retention affects model inference performance. Our observations reveal that the irregular sparsity introduced by unstructured retention breaks the vertical integrity of critical token refinement, hindering the accumulation of token information across layers and leading to severe performance degradation. Motivated by these findings, we propose VertiKV, a structured KV cache compression strategy that: (1) enforces vertical token consistency via Vertical-Aligned Structural Eviction (VASE), which partitions adjacent layers into minimal structural units (i.e., chunks) with adjustable depths and executes a token-level union within each chunk to align with the inherent vertical refinement process of token representations; and (2) enhances semantic consistency via Top-Down Information Enhancement (TDIE), which exploits the aggregated semantics of evicted tokens encoded in retained tokens and performs similarity-guided merging from upper aligned retained tokens within each chunk to fully utilize residual context. Extensive experiments on representative MLLMs across challenging multimodal long-context benchmarks demonstrate that our method achieves 1.2–1.9× decoding speedup and reduces KV cache memory by 80–95% while maintaining task performance.