Skip to content
Book Open access

Simulating Diverse User Behavioral Stereotypes for Evaluating Agentic Conversational Recommenders

Sep 2026 · Proceedings of the 20th ACM Conference on Recommender Systems · pp. 486-495 · 1 citation · 28 references

Abstract

Agentic Conversational Recommender Systems (ACRSs) are designed to recommend through multi-turn dialogue with users whose needs are not fully formed at the outset. However, their evaluation almost exclusively relies on user simulators that instantiate users with clear, pre-formed needs, reducing the interaction to a retrieval over attributes disclosed in the initial turns of the conversation. This covers only a narrow slice of the behaviors real users exhibit, and assessing the robustness and reliability of these systems requires simulated users that span a wider range. To this end, we introduce a stereotype-conditioned, open-weight user simulator that spans three behavioral stereotypes: Direct, Vague-Proactive, and Vague-Reactive. Benchmarking four state-of-the-art ACRSs across four e-commerce domains with our simulator, three findings emerge. First, under certain stereotypes, the user stops contributing new information about the target as turns accumulate. At the same time, the agent continues to act, a regime previously unobserved, which we name Unproductive Stagnation and formalize via Preference Coverage. Second, a systematic Robustness Gap emerges: as the simulated user shifts from decisive to passive, accuracy collapses while conversations grow longer. Third, accuracy degrades more sharply than Preference Coverage does, decomposing the gap into two separable capabilities current ACRSs lack: elicitation and retrieval, which current evaluation entangles in a single score. Our simulator makes this distinction reportable and gives the field a controllable axis along which elicitation and retrieval can be measured and compared.

Read PDF

Similar papers

Book Open access Sep 2026

A Design-Space Approach to Robust Agentic Conversational Recommendation

Agentic Conversational Recommender Systems (ACRSs), in which a large language model orchestrates dialogue, item retrieval, and tool use (such as querying item attributes or ranking candidate items), are advancing quickly, yet two problems prevent us from fully understanding their potential and limitations. First, progr...

Alessandro Francesco Maria Martina · 0 citations
#artificial intelligence Open access Sep 2026

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, ca...

Xin-Wei Yang, Ke-Long Mao, Yu-Dong Guo et al. · 1 citation
Preprint Aug 2026

Towards Faithful Simulation of Human Shopping Behavior

RecVerse is presented, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories and significantly outperforms existing baselines in both behavioral fidelity and intent consistency.

Jia-Kai Tang, Yan Mi, Jing Yu et al. · 0 citations
Review Open access Oct 2026

Personalization Matters: Long-Horizon Conversation Agent with User-Centric Information in Online Shopping Interactions

Personalized conversational shopping requires maintaining preference consistency over multi-turn interactions, where users reveal constraints gradually. Existing approaches often rely on static profiles and do not explicitly control long-horizon interaction behavior. We propose a multi-agent, multimodal Retrieval-Augme...

Re-Na Gao, Yue Dai, Hao Guan et al. · 1 citation
#artificial intelligence Preprint Sep 2026

A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation

A three-tier persona vector with 23 operationalized dimensions, demonstrating the model faithfully reproduces real-world difficulty distributions and seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices.

Rahul Khedar, Eshita, S. R. Thondapu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.