Speaking ITM’s Language: Query Reformulation and Temporal-MMR for Long-Video QA
Long-video question answering (QA) forces multimodal large language models (MLLMs) to work within a tight frame budget, so the choice of frames largely decides whether a question can be answered at all. The standard recipe scores every frame against the question with a pretrained image–text matching (ITM) model and keeps the top scorers. A fundamental mismatch underlies this recipe: ITM models are trained on short, concrete visual descriptions, while QA questions are interrogative and often involve abstract terms. Scored against the question alone, the ITM yields a near-random signal whenever the question is not a direct image–text match, such as one asking for the temporal order of scenes. In our LongVideoBench diagnostic analysis, the score collapses even on benchmark-provided answer-relevant frames, with more than half falling into a near-zero region —not because the encoder is faulty, but because it behaves exactly as it was trained to. We argue that this format mismatch should be absorbed at the two ends of the pipeline while the encoder itself stays frozen. On the input side, a type-conditioned routed pipeline reformulates each question into a single ITM-aligned description by selectively applying grounding, decomposition, and constrained synthesis. The ITM therefore receives exactly one description per frame, preserving the per-frame matching cost of a standard single-query baseline. On the output side, because the score distribution remains polarized and answer frames are scattered in time, we replace top- $K$ selection with a parameter-free Rosin threshold followed by a temporal maximal-marginal-relevance (MMR) step that uses frame positions alone. Across three benchmarks (LongVideoBench, Video-MME, MLVU) and and four backbones (Qwen2-VL, Qwen2.5-VL, LLaVA-OneVision, LLaVA-Video), the resulting training-free pipeline, RECAST, consistently outperforms recent frame-selection baselines without modifying the ITM encoder.