Skip to content

VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding

Sep 2026 · arXiv.org · Vol abs/2609.38413 · 0 citations · 68 references
Computer Science

TL;DR

VidHarness is proposed, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment as Monte Carlo tree search (MCTS).

Abstract

Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to $11.2$ points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness

LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reasoning, and memory. Recursive harness evolution can automate their design, but how it can reliably discover generalizable systems from limited, noisy, and repeatedly reuse...

Zi-Hao Lin, Hao-Yu Dong, Hao-Bo Xu et al. · 1 citation
Preprint Oct 2026

VideoEvoHarness: Self Evolving Agent Harness from Unlabeled Videos for Video Understanding

Video understanding agents rely on a harness to acquire evidence, use tools, and reason, but improving it requires substantial effort. Recent self evolution methods seek to optimize harnesses through execution feedback to reduce human intervention. However, existing methods often rely on annotated video questions and a...

Wei-Xin Xu, Yi-Fan Liu, Yi Wang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding

Video temporal grounding aims to localize events in videos from natural-language queries. For agents built around frozen video-language models, the harness determines how queries guide temporal predictions and how those predictions are refined. Manually refining these harnesses requires diagnosing grounding failures an...

Bin Luo, Yu-Huan Fan, Jia-Lin Guo et al. · 0 citations
Preprint Oct 2026

Rethinking Video Autoencoders for Interactive World Models

Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to...

Adrien Ramanana Rahary, Yu-Tong Bai, Nicolas Dufour et al. · 0 citations
Review Sep 2026

LVMH: A Meta-Optimization Harness for Long-Video Agent Memory

This work proposes Long Video Meta-memory Harness (LVMH), an automated multi-agent framework that formulates memory-policy development as multi-objective code search around a fixed answer model to guard against self-certification and development-set overfitting.

Li-Wei Che, Ben-Lin Liu, Dai Li et al. · 0 citations
#artificial intelligence Preprint Oct 2026

CASE: Cost-Aware Stopping for Efficient Long-Video Agents

Long-video agents can actively gather question-relevant evidence, but they typically leave a central decision implicit: when has the agent seen enough to answer? We propose CASE, a plug-in termination framework that frames this decision as policy-conditioned sequential stopping. At each causal checkpoint, CASE combines...

Yi-Ming Du, Cheng-Hao Liu, Zhiyuan Liu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.