This work proposes VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles, and introduces Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition.
Abstract
Long-term memory enables agents to accumulate information and reason across sessions, yet existing research primarily focuses on dyadic text or image-text conversations, leaving long-term memory for multi-party spoken conversations underexplored. This setting requires preserving conversational content, identifying participants across sessions, and retaining who speaks to whom. To this end, we propose VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles. We formulate retrieval as sequential decision-making, where an agent rewrites queries and selects retrieval tools and memory layers based on accumulated evidence to address information gaps. We further introduce Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage complementary evidence acquisition. We also construct VoxPolyBench to evaluate memory evolution, personalized answering, memory retrieval and reasoning, and interaction reasoning and attribution in multi-party spoken conversations. VoxPolyMem achieves an overall score of 85.0 on VoxPolyBench, surpassing the strongest evaluated baseline by 23.6 points. On Mem-Gallery and H2HMem-Multi, it scores 89.6 and 74.4, respectively, exceeding the strongest evaluated public memory baselines by over 8 points each. These results highlight its potential for persistent, personalized assistance in multi-party multimodal interactions. Code and datasets are available at https://voxpolymem.github.io/VoxPolyBench/demo/
SpeakerMem-R1 is proposed: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time, and ablations show that the verbatim and structured tracks, as well as person-...
Hao-Bo Zheng, Tan Tang, Yan Chen et al.· arXiv.org· 0 citations
This work investigates whether memory interference originates mainly from memory retrieval or from the accumulation of competing fact versions added during memory updates, and evaluates how memory-write policies influence memory retrieval behavior later on.
Erica Butts, Salam Daher· Proceedings of the 26th ACM...· 0 citations
This work proposes a novel Human-profile Enhanced Retrieval Optimization framework for long-term agent memory (HERO), which converts the dialogue history into a traceable heterogeneous memory graph that preserves raw dialogue text as evidence for reasoning, thereby mitigating information loss.
Yuanhua Lin, Yile Li, Zhi-Yuan Zhao et al.· 0 citations
CUE-Mem provides a testbed for memory systems that selectively retain, retrieve, and use subtle multimodal evidence, and provides a testbed for memory systems that selectively retain, retrieve, and use subtle multimodal evidence.
Yu-Lin Hu, Yan-Yan Zhao, Zi-Mo Long et al.· 0 citations
This work proposes MemLoc, a unified Retrieve-Localize-Generate framework for long-term conversational memory QA, and introduces a reasoning-based evidence locator trained with Self-reflective Hint Policy Optimization, which performs progressive refinement by extracting query-relevant fragments within memory units to s...
Yi-Fan Wang, Xin-Kui Lin, Yong-Xiu Xu et al.· 0 citations
A lightweight adapter, DiaRelay, is proposed to enable LLMs to explicitly maintain a dialogue-level memory for accurate ERC, based on LoRA, which can achieve SOTA weighted F1 and accuracy on MELD while obtaining competitive results on IEMOCAP with only an extra 7.1M trainable parameters.
Zi-Hao Zhou, Bin Yang, Jinghui Qin et al.· 0 citations