Skip to content
Preprint

UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents

Oct 2026 · 0 citations · 44 references
Computer Science

TL;DR

This work introduces UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types, and proposes On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts.

Abstract

Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users'behavioral responses over long interactions. Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation and offline validation against production A/B results. We introduce UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. UserArena pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes. To improve simulator fidelity over these long-horizon interactions, we further propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. Experiments show that OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol and most closely reproduces the production policy effect in offline simulation.

View source

Similar papers

#artificial intelligence Open access Sep 2026

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, ca...

Xin-Wei Yang, Ke-Long Mao, Yu-Dong Guo et al. · 1 citation
Review Open access Oct 2026

Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augme...

Re-Na Gao, Yue Dai, Hao Guan et al. · 1 citation
#machine learning Preprint Sep 2026

PsyBer-Agent: A Psychology-Driven User Agent for Recommender Systems

Evaluating recommender systems before deployment requires simulators that remain realistic over long, closed-loop interaction cycles. Current LLM-based agents generally function as surface imitators that generate realistic short-term responses but fail to capture the underlying user dynamics such as fatigue, affect, an...

Chong Zhang, Qi-Jie Shen, Hai-Chao Zhang et al. · 0 citations

SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling

Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. However, building them requires access to large-scale, semantically faithful, fine-grained online user trajectories. These data are difficult to obtain because proprieta...

Yunan Lu, Shuang-Yang Xie, Meghna Allamudi et al. · 0 citations
#artificial intelligence Review Sep 2026

Automatic multimodal UX improvement recommendations from LLM agent user simulations

AMUSER is presented, a multimodal framework which simulates user behaviour and automatically generates prioritised UX improvement recommendations from resulting data and suggests an asymmetric role of multimodality: visual access during simulation improves recommendations through richer traces, while providing visual i...

Anurupa Chowdhury, Bin Wu, Hossein A. Rahmani et al. · 0 citations
Sep 2026

AdaptArena: Evaluating Test-Time Personalization of Web Agents

It is found that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference.

Dong-Chan Shin, Xing Han Lù, Jiaqi Deng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.