Skip to content
Review

LVMH: A Meta-Optimization Harness for Long-Video Agent Memory

Sep 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work proposes Long Video Meta-memory Harness (LVMH), an automated multi-agent framework that formulates memory-policy development as multi-objective code search around a fixed answer model to guard against self-certification and development-set overfitting.

Abstract

Long-video agents must map hours or days of multimodal evidence into a bounded context while respecting token and latency budgets, making memory design a context-management challenge. Existing memory systems largely rely on hand-designed policies, leaving open how to jointly optimize memory design for performance and efficiency. We propose Long Video Meta-memory Harness (LVMH), an automated multi-agent framework that formulates memory-policy development as multi-objective code search around a fixed answer model. To guard against self-certification and development-set overfitting, LVMH separates implementation from independent review and enforces protected observation boundaries. Persistent meta-memory records past proposals, traces, costs, and failures to guide subsequent policy revisions. Through this harness, we discover Self-Evolving Evidence Memory (SEEM), a"text-first, vision-second"policy that answers from compact structured event memory and evidence specialists, consulting localized frames through a selective visual gate. The policy evolved on EgoLifeQA generalizes to unseen benchmarks, outperforming Meta-Harness on both Ego-R1 Bench and Video-MME (L) with GPT-5 mini. Across the three benchmarks, SEEM with GPT-5 achieves 65.7% average accuracy, surpassing the previous state of the art by 7.7 percentage points with approximately 32x lower average evaluation token use and 91x shorter average response time.

View source

Similar papers

Just-In-Time Agent Memory with Runtime Agentic Research

Just-In-Time Agent Memory (JAM), a trainable framework for query-conditioned context construction at runtime, is proposed, where it achieves stronger task performance than AOT-style memory systems while remaining substantially more efficient than prior trained agentic memory approaches.

Bing-Yu Yan, Chao-Fan Li, Hong-Jin Qian et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning

This work introduces a unified formulation of context optimization and shows that an agent memory system update can be interpreted as an optimization update procedure over the model's context, and proposes GraphMemory, a lightweight graph-based memory that accumulates, refines, organizes, and connects reusable strategi...

Yehya Farhat, Michael Desmond, Anastasios Kyrillidis · 0 citations
#artificial intelligence Preprint Oct 2026

All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents

All-in-One Memory (AiO-Mem), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action in a unified reward space, consistently improves answer accuracy over both static memory systems and RL-trained memory agents.

Bing-Bing Wang, Jing-Jie Lin, Wei-Ming Qiao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

MemAgent: Learning to Manage Heterogeneous Memory Providers for LLM Agents

MemAgent, featuring a content-aware routing architecture and a training-data synthesis pipeline, is proposed, featuring a content-aware routing architecture and a training-data synthesis pipeline that improves average accuracy and outperforms every individual memory method across all three benchmarks.

Yong-Xian Wei, Yi-Lin Zhao, Run-Xi Cheng et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Interactive Memory Learning for Long-Term Conversations

InteraCtive Memory Learning is proposed, a multi-agent framework that transforms the memory mechanism from a passive archive into a learnable, interactive memory policy that significantly outperforms strong baselines.

Cai Ke, Jiang-Yue Yan, Han Zhang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

CASE: Cost-Aware Stopping for Efficient Long-Video Agents

Long-video agents can actively gather question-relevant evidence, but they typically leave a central decision implicit: when has the agent seen enough to answer? We propose CASE, a plug-in termination framework that frames this decision as policy-conditioned sequential stopping. At each causal checkpoint, CASE combines...

Yi-Ming Du, Cheng-Hao Liu, Zhiyuan Liu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.