Preprint

VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models

Guo-Yang Xu Hao Chen

Aug 2026 · 2 citations · ⚡ 1 influential · 22 references

Computer Science

TL;DR

VideoHarness-RSI is introduced, a controlled framework that recursively searches executable context constructors around a frozen VLM while keeping the answering model and interface fixed and establish executable context construction as a distinct optimization layer and provide an auditable baseline for studying harness discovery, transfer, and efficiency around frozen VLMs.

Abstract

Long-video understanding depends not only on the capability of a vision-language model (VLM), but also on how its limited context is constructed from a much longer video. Existing systems typically introduce hand-designed sampling, retrieval, memory, or agentic control strategies, making the context-construction program itself difficult to study as an independent optimization target. We introduce VideoHarness-RSI, a controlled framework that recursively searches executable context constructors around a frozen VLM while keeping the answering model and interface fixed. We study this baseline under complementary weak- and strong-initialization regimes. From a weak uniform constructor, recursive search progressively discovers more structured context-construction programs; from a stronger AKS harness, the same process further advances an already competitive hand-crafted frontier. The resulting harness retains its advantage under a matched cumulative visual-token control and transfers directly to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide an auditable baseline for studying harness discovery, transfer, and efficiency around frozen VLMs.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

SimpleMemVLA, a VLA without a dedicated memory module that uses the backbone's native video context directly as memory, achieves state-of-the-art results on four memory benchmarks without loss on general-purpose control, and with the same backbone and training setup it outperforms retrieval, compression and recurrent-s...

C. Yin, Wang Xu, Jun-Peng Yang et al. · 2 citations

Preprint Aug 2026

LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

LongVU-TTT is introduced, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM, and is stronger than attention- and fixed-state recurrent resamplers across three benchmarks.

Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase et al. · 0 citations

Preprint Aug 2026

Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding

Clue-OPSD, a clue-privileged on-policy self-distillation framework for long-video understanding that uses clue intervals as privileged supervision without relying on ground-truth answer labels, while requiring no clue annotations or additional modules at inference time is introduced.

Kai-Shen Wang, Dong-Di Zhao, Yijun Liang et al. · 6 citations

#computer vision Review Sep 2026

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count, and organize methods by the pipeline stage at which they act.

Killian Steunou, Yannis Tevissen, M. El Yacoubi · 0 citations

Jul 2026

Wonder: Video World Model Done Better

This work introduces a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues, allowing the model to interpret camera motion directly as visual evidence, regardless of actual context length.

Jiacong Xu, Hanwen Jiang, Zhixin Shu et al. · arXiv.org · 5 citations · ⚡1

Preprint Aug 2026

Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

VideoRouter (VR) is proposed that rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames, and introduces a verification-guided router to determine which view is better supported by the selected evidence and select the final answer.

Ziling Huang, Yuki M. Asano, Shin'ichi Satoh · 0 citations