Question-in-the-Loop: Interleaved Query Tokens as Visual Summarizers for Video LLMs
Abstract
Extending vision-language models (VLMs) to long-form video understanding is bottlenecked by the massive volume of visual tokens that incur quadratic prefill cost in the LLM decoder. Existing compression methods either discard tokens before the question is known (query-agnostic) or passively observe post-hoc attention patterns (passive query-aware), risking premature loss of question-relevant details. We propose Question-in-the-Loop (QitL), which actively interrogates the visual stream by structurally interleaving the actual text query tokens between temporal chunks of video frames inside the LLM. Each query copy uses causal self-attention but is visually restricted to its assigned chunk, simultaneously serving three roles: a pruning guide that scores visual token relevance, a local attention sink that mitigates attention dilution in long sequences, and a persistent summary that carries compressed, question-conditioned information into deeper layers. A lightweight prune-merge step then removes low-relevance tokens while folding residual information into the summaries. We further provide empirical evidence that shallow-layer attention is semantically uncalibrated for pruning and that local masking accelerates the emergence of meaningful cross-modal attention, which motivates our aggregation boundary design. Entirely training-free, QitL requires no learnable parameters or weight modifications. Experiments on Video-MME, MLVU, LongVideoBench, and MVBench demonstrate competitive accuracy–efficiency trade-offs among the reported training-free baselines. Code will be released at https://github.com/NihiI-obstat/QitL.