EVA: Efficient Token Compression Co-Optimization With Heterogeneous CIM Architecture for Video LLM Acceleration
Abstract
Video large language models (VLLMs) enable powerful multimodal reasoning but face severe efficiency challenges on edge devices due to the massive computational and memory demands caused by lengthy video-token sequences. Existing methods struggle to efficiently compress spatiotemporally redundant tokens while minimizing DRAM access overhead during inference. In this work, we propose EVA, a co-designed algorithm–hardware framework for accelerating VLLMs. At the algorithm level, we introduce an efficient training-free token compression (ETC) method that combines greedy temporal segmentation (GTS) to adaptively partition frames by content similarity, with diversity spatiotemporal compression (DSC) to retain semantically rich tokens from both static and dynamic regions. The method is plug-and-play, requires no retraining, and employs sign similarity to enable hardware-friendly computing at scale. At the hardware level, we design a heterogeneous accelerator integrating a lightweight token compression engine (TCE), a computing-in-memory (CIM) engine for in situ execution of linear layers, and a reconfigurable digital attention engine with hardware-specialized attention computation and an interleaved pipeline dataflow. Across multiple models and video benchmarks, EVA preserves accuracy under aggressive token reduction and delivers substantial efficiency gains. Notably, on LLaVA-OneVision-7B, EVA compresses 90% of video tokens while maintaining 97.8% of the original accuracy. EVA achieves up to <inline-formula> <tex-math notation="LaTeX">$10.3\times $ </tex-math></inline-formula> speedup and <inline-formula> <tex-math notation="LaTeX">$47.4\times $ </tex-math></inline-formula> energy reduction compared to GPU, and achieves up to <inline-formula> <tex-math notation="LaTeX">$3.8\times $ </tex-math></inline-formula> speedup over prior specialized accelerators, demonstrating a practical path toward scalable VLLM inference on resource-constrained devices.
