DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection
TL;DR
Inspired by dual-coding theory, this work proposes a memory architecture that uses parallel visual and verbal codes, which it calls DualMem, and views this as a step towards memory systems that preserve a richer record of agents'observations.
Abstract
Agent benchmarks for measuring memory largely study textual cases, in which information is deliberately extracted from the environment, written down, and then later retrieved. In other words, they assess what agents elected to record, not what they happened to see. We introduce DMV-Bench (code: https://github.com/yyyujintang/DMV-Bench), the first interactive benchmark for visual memory in multimodal agents, to study this often-neglected property. DMV-Bench is built on (1) a controlled home-furnishing e-commerce environment, supported by a catalog of 1,000 product variants, and (2) a text-leakage contract which ensures that the primary discriminative signal of each task is solely in the pixels. In DMV-Bench, agents undergo chains of autonomous shopping sessions in which every visited product image carries a unique, pre-rendered incidental cue that the agent is later asked to recall. We show that conventional solutions struggle with this task. Inspired by dual-coding theory, we propose a memory architecture that uses parallel visual and verbal codes, which we call DualMem. On DMV-Bench, DualMem outperforms a caption-only baseline and three recent multimodal agent-memory systems across multi-session chain lengths on multiple models. These gains persist even adjusting for memory-bank size and encoding-position bias. Further experiments also reveal an asymmetric division of labor between the two codes; a weighted coding scheme is often strongest. We view this as a step towards memory systems that preserve a richer record of agents'observations.
