This work introduces UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types, and proposes On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts.
Abstract
Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users'behavioral responses over long interactions. Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation and offline validation against production A/B results. We introduce UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. UserArena pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes. To improve simulator fidelity over these long-horizon interactions, we further propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. Experiments show that OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol and most closely reproduces the production policy effect in offline simulation.
This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, ca...
Xin-Wei Yang, Ke-Long Mao, Yu-Dong Guo et al.· Conference on Empirical Meth...· 1 citation
Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augme...
Re-Na Gao, Yue Dai, Hao Guan et al.· Findings of the Association...· 1 citation
Evaluating recommender systems before deployment requires simulators that remain realistic over long, closed-loop interaction cycles. Current LLM-based agents generally function as surface imitators that generate realistic short-term responses but fail to capture the underlying user dynamics such as fatigue, affect, an...
Chong Zhang, Qi-Jie Shen, Hai-Chao Zhang et al.· 0 citations
Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. However, building them requires access to large-scale, semantically faithful, fine-grained online user trajectories. These data are difficult to obtain because proprieta...
Yunan Lu, Shuang-Yang Xie, Meghna Allamudi et al.· arXiv.org· 0 citations
AMUSER is presented, a multimodal framework which simulates user behaviour and automatically generates prioritised UX improvement recommendations from resulting data and suggests an asymmetric role of multimodality: visual access during simulation improves recommendations through richer traces, while providing visual i...
Anurupa Chowdhury, Bin Wu, Hossein A. Rahmani et al.· arXiv.org· 0 citations
It is found that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference.
Dong-Chan Shin, Xing Han Lù, Jiaqi Deng et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.