Jul 2026· Annual International ACM SIGIR Conference on Research and Development in Information Retrieval· 0 citations· 67 references
Computer Science
TL;DR
Inspired by Multiple-Trace Theory in cognitive psychology, this work revisit long video understanding from a probe-echo perspective, in which human episodic memories are activated and integrated in parallel, and proposes ProEchoMem, a cognitive-inspired framework that simulates the probe-echo mechanism.
Abstract
Large vision-language models (LVLMs) have shown significant progress in video understanding, but they struggle to scale to long videos due to limited context windows. Existing methods reduce input dimensionality via frame sampling and feature compression, yet discard details and incur high computational cost for post-training. In contrast, retrieval-augmented generation (RAG) that indexes long videos for query retrieval and memory-based methods that maintain evolving long-term stores, offer a lighter and deployment-friendly solution. Nevertheless, they rely on shallow retrieval that selects only top-ranked segments and fails to integrate information across multiple relevant video episodes. Inspired by Multiple-Trace Theory in cognitive psychology, we revisit long video understanding from a probe-echo perspective, in which human episodic memories are activated and integrated in parallel. Building on this insight, we propose ProEchoMem, a cognitive-inspired framework that simulates the probe-echo mechanism: (1) Incremental Episodic Memory Construction builds structured knowledge graphs from video streams; (2) Probe-Driven Memory Activation generates probe signals from user queries to activate all stored traces simultaneously; (3) Memory Echo Synthesis integrates activated traces into a coherent and structured memory echo. Experiments on LongerVideos, LVBench, and cross-domain settings demonstrate the effectiveness of ProEchoMem, with multi-trace probing achieving up to 14.2% higher relevance and ablation studies validating the contribution of each module. The code is available at https://github.com/Applied-Machine-Learning-Lab/SIGIR26_ProEchoMem
Long-term video object segmentation (VOS) remains challenging due to error accumulation under extended occlusions, re-appearance, and scene changes. Although SAM2 provides strong zero-shot performance, its streaming memory can amplify drift over long horizons when recent, unreliable predictions dominate the memory state. We propose SAM2Dual, a training-free, plug-and-play inference-time enhancement that improves long-video robustness without updating model weights. SAM2Dual introduces a Dual Memory design that explicitly separates (i) short-term memory for rapid local adaptation and (ii) long-term memory built via interval-based sampling to preserve global identity cues, combined through a gated fusion strategy. In addition, we present Text-Aware Memory (TAM), which extracts a compact word-level cue from early frames and uses text embeddings to reweight memory contributions based on semantic compatibility, supporting identity preservation when visual evidence becomes weak or ambiguous. Across long-term benchmarks, SAM2Dual consistently improves stability on long videos, raising J&F from 49.33 to 50.65 on MOSEv2 and achieving consistent gains on LVOSv2.
Long-video understanding depends critically on how a limited model context is constructed from a much longer video. Existing approaches improve this process through compression, retrieval, memory, and agentic evidence acquisition, but these mechanisms are typically introduced as part of a manually designed inference system or optimized together with other components. This makes it difficult to isolate a simpler question: how much can be gained by improving the executable context-construction program alone? We study this question through VIDEOHARNESS-RSI, a controlled baseline for recursively searching executable context constructors around a frozen vision-language model (VLM). An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed and evaluated end to end before successful variants are retained for further search. This makes long-video understanding a controlled instance of automated harness design: the searchable object is executable program structure, while the answering model and interface remain fixed. Starting from uniform sampling, recursive harness search consistently finds room for improvement and surpasses several weaker hand-crafted baselines. Starting instead from a stronger hand-crafted baseline, the same RSI process yields a further improvement. The selected harness also transfers to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide a reproducible baseline for studying harness discovery and transfer around frozen VLMs.
This paper introduces TinyMem, a model built upon compact multimodal memory for long-form video action detection that outperforms a range of state-of-the-art models on AVA v2.2 while using 5 times fewer memory tokens than the baseline with dense visual memory embeddings.
Rui Tian, Qi Dai, Hang-Rui Hu et al.· IEEE Transactions on Pattern...· 0 citations
This work forms an episodic multi-key representation that enables precise retrieval of fine-grained memories through a simple key-matching mechanism, and introduces a neighbor filtering mechanism to capture broader semantic context without the massive computational overhead of global memory construction.
Yeeun Choi, Youngbeom Yoo, Joon-Young Lee et al.· 0 citations
Streaming video understanding requires multimodal large language models (MLLMs) to preserve relevant evidence from continuously evolving streams under strict causality and bounded memory. Yet existing paradigms remain limited: model-based methods require intrusive backbone updates, while memory-based methods expend substantial visual-encoding computation on temporally redundant content and rely on rigid access to visual history. To address these limitations, we introduce StreamFlow, an efficient visual memory framework that enables dynamic, on-demand access to historical visual information. StreamFlow combines a lightweight, dynamics-aware mid-term memory that filters temporal redundancy before visual encoding with a latent long-term memory that consolidates historical video content into visual latents accessible to subsequent reasoning. During generation, an attention-guided retrieval mechanism injects relevant visual latents when the model's reliance on visual evidence weakens. StreamFlow achieves state-of-the-art streaming video understanding performance, reaching 67.73% overall accuracy on StreamingBench, while also delivering strong performance on offline long-video benchmarks. Relative to the vanilla setting, it improves the visual attention score (VAS) by 59.1% while reducing end-to-end latency and peak memory by 50.4% and 21.1%, respectively, enabling more visually grounded and efficient reasoning.
Muxin Fu, Yifan Zhang, Wentao Zhang et al.· 0 citations
Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.
Xinru Jiang, Lin Zhao, Xi Xiao et al.· 0 citations
