Skip to content

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Sep 2026 · arXiv.org · Vol abs/2609.32182 · 0 citations · 48 references
Computer Science

TL;DR

KeyRec is a training-free framework for constructing bounded visual memory that preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank and achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual-token budget.

Abstract

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose \textbf{KeyRec}, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

View source

Similar papers

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs) that separates video ingestion from query answering, consistently outperforms frozen baselines at every evaluated visual budget.

Si-Ru Zhong, Qiong-Yan Wang, Xiao-Hui Lv et al. · 0 citations
Preprint Aug 2026

Dynamic Hub-and-Spoke Memory for Streaming Video Understanding

Dynamic Hub-and-Spoke Memory is proposed, a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception in streaming video understanding.

Xin-Ru Jiang, Lin Zhao, Xi Xiao et al. · 4 citations
#artificial intelligence Preprint Oct 2026

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed, and support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Hao-Zhe Chi, Song Jin, Yang Jin et al. · 1 citation
Conference Open access Sep 2026

CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

CoFiE is proposed, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill that outperforms strong open-source multimodal models while improving end-to-end infer...

Jing-Chi Jiang, Yi-Ran Ling, Ruo-Nan Li et al. · 1 citation
#computer vision Review Sep 2026

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count, and organize methods by the pipeline stage at which they act.

Killian Steunou, Yannis Tevissen, M. El Yacoubi · 0 citations
#machine learning Preprint Sep 2026

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

SimpleMemVLA, a VLA without a dedicated memory module that uses the backbone's native video context directly as memory, achieves state-of-the-art results on four memory benchmarks without loss on general-purpose control, and with the same backbone and training setup it outperforms retrieval, compression and recurrent-s...

C. Yin, Wang Xu, Jun-Peng Yang et al. · 4 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.