Abstract
Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-level decision support: users reveal and revise constraints, coordinate multiple goals, and expect product-grounded recommendations over a full conversation. Existing benchmarks are mostly outcome-oriented or execution-oriented, leaving this evolving decision process under-evaluated. We introduce REALWORLDSHOP, a benchmark built on 3.28M grounded products, structured shopping episodes, a profile-grounded and actioncontrolled user simulator, and role-play evaluation. Our analysis shows that current systems produce locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially under ambiguous intent, bundle, and multi-intent scenarios. We further propose REALSHOP_AGENT, an executable session-control framework with explicit state management, shopping-flow control, catalog-grounded retrieval, and runtime guards. Experiments show that REALSHOP_AGENT consistently outperforms strong baselines on REALWORLDSHOP.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, X., Mao, K., Guo, Y., Xu, S., Gu, S., Huang, C., & Lei, W. (2026). RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce. https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce
MLA 9
Yang, Xinwei, et al. "RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce." https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce.
Chicago (author–date)
Yang, Xinwei, Kelong Mao, Yudong Guo, Sulong Xu, Simiu Gu, Chen Huang, and Wenqiang Lei. 2026. "RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce." https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce.
Harvard
Yang, X., Mao, K., Guo, Y., Xu, S., Gu, S., Huang, C. and Lei, W. (2026) 'RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce', Available at: https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce.
Vancouver
Yang X, Mao K, Guo Y, Xu S, Gu S, Huang C, et al. RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce. https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce
IEEE
X. Yang, K. Mao, Y. Guo, S. Xu, S. Gu, C. Huang, and W. Lei, "RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce," https://omanscience.com/en/articles/realworldshop-benchmarking-and-improving-conversational-shopping-agents-in-real-world-e-commerce.