Skip to content

MEDR: Query-Independent Frame Selection via Multi-Signal Event Modeling and Dynamic Rescoring

Aug 2026 · arXiv.org · Vol abs/2608.15058 · 1 citation · 28 references
Computer Science

TL;DR

This work proposes Multi-Signal Event Modeling and Dynamic Rescoring (MEDR), a training-free and query-independent frame selection method that preserves the reusability of a fixed visual input while improving the coverage of informative events beyond uniform sampling.

Abstract

Frame selection is a fundamental component of multimodal large language models, enabling long videos to be processed under limited visual-token and computational budgets. Uniform sampling preserves temporal coverage but may miss informative content that appears only briefly. To alleviate this limitation, query-dependent methods can retrieve question-relevant frames. However, because the selected frames depend on the current question, the same visual input cannot be directly shared across different questions, and frame selection must be repeated in multi-turn video dialogue. This motivates us to seek a query-independent frame selection method that preserves the reusability of a fixed visual input while improving the coverage of informative events beyond uniform sampling. We propose Multi-Signal Event Modeling and Dynamic Rescoring (MEDR), a training-free and query-independent frame selection method. Multi-Signal Event Modeling organizes complementary visual, motion, and text signals into signal-specific temporal events. Dynamic Rescoring then iteratively reevaluates each candidate relative to the current selected set, updating its score according to frame-level signal strength, additional event coverage, and balanced temporal coverage. The resulting fixed frame set is constructed without observing the query and can be reused across different questions. Experiments on Video-MME and LongVideoBench demonstrate that MEDR achieves competitive performance across two video MLLMs. Under the 64-frame budget, MEDR improves Video-MME accuracy over the strongest evaluated query-independent baseline by 1.33-1.78 percentage points. We further evaluate the reuse of a fixed frame set in multi-question video dialogue.

View source

Similar papers

Query-aligned video frame selection for long video understanding

This work introduces a frame-selection method designed specifically for multiple-choice questions, and is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question.

Md. Safayet Islam, Dilip Sarkar, Liang-Yu Liang · 1 citation
Preprint Oct 2026

EVISTA: Unified Multimodal Video Search with Evidence-Aware Temporal Path Optimization

In the era of big data, precise event retrieval from large video collections has become increasingly important and challenging. In particular, locating multiple events from the same video in their temporal order is difficult when different events are described by different types of evidence, such as visual content, on-...

M. Tran, V. Vo, M. Nguyen et al. · 0 citations
Preprint Oct 2026

A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding

Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning. Existing query-aware methods typically rely on heuristic pipelines: they assign one-dimensional query-c...

Zi-Long Zhao, Yu-Hang Liu · 0 citations
#artificial intelligence Preprint Nov 2026

Reason First, Retrieve What Matters: Hypothesis-Guided Keyframe Selection for Long Video Understanding

Long video understanding requires models to identify and reason over sparse, temporally distributed evidence within limited frame budgets. Existing keyframe selection methods typically retrieve query-relevant frames from a densely sampled candidate set before inference. However, global image-text matching incurs unnece...

Yi-Song Hu, Jun-Bo Wang, Yu-Ke Li et al. · 0 citations
Preprint Oct 2026

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most o...

Yu-Hao Qin, Jun-Bo Wang, Yu-Ke Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.