Sep 2026· Conference on Empirical Methods in Natural Language Processing· Vol abs/2609.38974· 1 citation· 36 references
Computer Science
TL;DR
This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards.
Abstract
Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.
Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main cha...
Ju-Li Huang, Hanna Clay, Sajjad Beygi et al.· 0 citations
RecVerse is presented, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories and significantly outperforms existing baselines in both behavioral fidelity and intent consistency.
Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next-step guidance into a coherent recommendation experience. This changes the unit of evaluation: a fluent response can still fail by ignoring shopper cont...
Weimin Lyu, Chen Luo, Guangrui Li et al.· 0 citations
A framework for evaluating and improving a large-scale, multi-agent shopping assistant in production in production is presented, and lessons from its use are reported.
Kasra Hosseini, Wen-Sen Cheng, Marco-Andrea Buchmann et al.· 0 citations
This work introduces rubric-based preference reasoning, which uses automated rubrics to guide the model toward producing explicit preference states, and proposes a preference-conditioned optimization to align response generation with dual objectives: user preference satisfaction and situation consistency.
Dongding Lin, Jian Wang, Xiao-Ya Zhao et al.· 0 citations
Agentic Conversational Recommender Systems (ACRSs) are designed to recommend through multi-turn dialogue with users whose needs are not fully formed at the outset. However, their evaluation almost exclusively relies on user simulators that instantiate users with clear, pre-formed needs, reducing the interaction to a re...
Alessandro Petruzzelli, Alessandro Francesco Maria Martina, C. Musto et al.· Proceedings of the 20th ACM...· 1 citation