Skip to content

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.38974 · 1 citation · 36 references
Computer Science

TL;DR

This work introduces REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation, and proposes REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards.

Abstract

Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.

Read PDF

Similar papers

Preprint Aug 2026

MACS: A Hybrid Multi-Agent Framework for Reliable Conversational E-Commerce Recommendation

Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main cha...

Ju-Li Huang, Hanna Clay, Sajjad Beygi et al. · 0 citations
Preprint Aug 2026

Towards Faithful Simulation of Human Shopping Behavior

RecVerse is presented, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories and significantly outperforms existing baselines in both behavioral fidelity and intent consistency.

Jia-Kai Tang, Yan Mi, Jing Yu et al. · 0 citations
Preprint Aug 2026

PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next-step guidance into a coherent recommendation experience. This changes the unit of evaluation: a fluent response can still fail by ignoring shopper cont...

Weimin Lyu, Chen Luo, Guangrui Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignment

This work introduces rubric-based preference reasoning, which uses automated rubrics to guide the model toward producing explicit preference states, and proposes a preference-conditioned optimization to align response generation with dual objectives: user preference satisfaction and situation consistency.

Dongding Lin, Jian Wang, Xiao-Ya Zhao et al. · 0 citations
Book Open access Sep 2026

Simulating Diverse User Behavioral Stereotypes for Evaluating Agentic Conversational Recommenders

Agentic Conversational Recommender Systems (ACRSs) are designed to recommend through multi-turn dialogue with users whose needs are not fully formed at the outset. However, their evaluation almost exclusively relies on user simulators that instantiate users with clear, pre-formed needs, reducing the interaction to a re...

Alessandro Petruzzelli, Alessandro Francesco Maria Martina, C. Musto et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.