Skip to content
Preprint

Profile-Driven User Simulation via Reinforcement Learning with Hierarchical Rewards for Task-Oriented Dialogues

Oct 2026 · 0 citations · 71 references
Computer Science

TL;DR

ProfileSim is proposed, a profile-driven user simulation framework for financial marketing outbound calls that achieves the highest colloquial naturalness, the strongest user profile adherence and the lowest agent score on long-tail intents among all evaluated user generators, while producing substantially shorter user utterances at comparable business compliance.

Abstract

Building Task-Oriented Dialogue (TOD) systems for high-value scenarios such as financial marketing requires large volumes of domain-specific dialogue data, which privacy regulations, annotation costs, and the need for expert labor make hard to obtain. User simulation offers a remedy, but existing methods target general-domain dialogue and exhibit two key limitations: insufficient modeling of domain-specific business processes and compliance constraints, and inadequate capture of the colloquial, spoken-language character of voice interactions. We propose ProfileSim, a profile-driven user simulation framework for financial marketing outbound calls. It jointly models user profiles---demographics and Big Five personality traits---and business profiles---intents reverse-engineered from real call logs---enabling fine-grained control over simulated behavior. The base model is fine-tuned on real voice dialogue data and refined by Group Relative Policy Optimization (GRPO) with hierarchical rewards at the dialogue and utterance levels. Paired with a customer service agent integrating Standard Operating Procedure (SOP) constraints and Retrieval-Augmented Generation (RAG), ProfileSim achieves the highest colloquial naturalness, the strongest user profile adherence and the lowest agent score on long-tail intents among all evaluated user generators, while producing substantially shorter user utterances at comparable business compliance. The generated dialogue data is designed to support intent recognition training, agent evaluation, and rapid cold-start deployment.

View source

Similar papers

#artificial intelligence Conference Open access Sep 2026

OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue

This work proposes OSPD, an on-policy self-distillation framework where the same model serves as both teacher and student under asymmetric information: the teacher receives a complete character profile while the student sees only a brief summary, and the student generates trajectories from its own policy.

Rui Xu, Yikai Zhang, Ai-Li Chen et al. · 0 citations
Sep 2026

TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

TRACER is proposed, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones, and the Dynamic Marketing Benchmark is introduced, revealing that higher response quality does not necessarily correspond to higher conversion rates.

Ge Chen, Ruo-Tong Pan, Zhi-Rui Yang et al. · 0 citations
Open access Oct 2026

Personalized dialogue generation based on social game data collection and two-stage response generation model

The development of artificial intelligence has promoted the widespread application of human-machine dialogue systems, and users urgently need personalized and emotionally resonant interactions. However, the scarcity of high-quality data and information hinders the generation of role consistent dialogues. To this end, t...

Zhi-Yi He, Yi-Han Wang, Han Shi · 0 citations

StateTree: Enhancing Long-Term Dialogue Reasoning via Reinforcement Learning

Large language models deployed as personalized assistants must reason over long, evolving interaction histories. However, in long-term dialogue reasoning, relevant evidence is scattered across sessions, preferences may be revised over time, and standard long-context training fails to address these challenges under data...

Na-En Xu, Wan-Qing Cui, Yi-Bo Hu et al. · 0 citations
#small language model Book Open access Nov 2026

Persona-Isolated Evaluation of LLM-Based Dialogue Agents: Measuring History-Conditioned Response Variation

Adaptive LLM-based agents can appear to adjust not only their answer content but also their response structure according to prior dialogue and user-stated preferences. This paper treats this phenomenon as a measurement problem, not as an attempt to reproduce naturalistic long-term human–agent interaction. We ask where...

Satoshi Koda, Hiroshi Someya, K. Kuroda · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.