This work investigates a method for automatically detecting inconsistencies in Task-Oriented Dialogues by conceptualizing TODs as a Constraint Satisfaction Problem (CSP), where variables represent dialogue segments referencing the conversational domain and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge.
Abstract
Generative AI is profoundly transforming the core technologies behind conversational systems, shifting from component-based to end-to-end approaches. However, Large Language Models (LLMs) may still generate inconsistencies, a critical issue particularly in Task-Oriented Dialogues (TODs), where system responses must strictly adhere to information from a domain knowledge base (e.g., restaurants in a city). A single hallucination (e.g., suggesting a non-existent restaurant) can lead to severe task failures. We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem (CSP), where variables represent dialogue segments referencing the conversational domain, and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge. We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions. By comparing the target dialogue with valid variable assignments, we can detect inconsistencies and suggest minimal changes to ensure dialogue consistency. We demonstrate the high accuracy of the CSP-based approach in detecting inconsistencies, and provide a detailed analysis of our findings.
Traditional conversational Question Answering (QA) systems focus on isolated information retrieval and lack the ability to maintain goal persistence across multi-turn interactions. Current Retrieval-Augmented Generation (RAG) approaches operate passively, retrieving information turn-by-turn without tracking whether the overall task objective has been achieved, and are typically evaluated using linguistic similarity metrics rather than functional task outcomes. We introduce GSC-QA (Goal-based Sequential Conversation QA), a framework that integrates three complementary components into a unified enterprise dialogue architecture: 1) persistent Goal Tracking Module (GTM) that maintains task state across turns in external databases (Redis + PostgreSQL), decoupled from the LLM context window; 2) Expert Instruction Layer that injects domain-curated procedural constraints into every response generation step; and 3) proactive slot detection that identifies and elicits missing required information before proceeding. Unlike prior task-oriented systems that assume static ontologies or session-scoped memory, GSC-QA combines retrieval, instruction enforcement, and goal persistence in a single coordinated loop built on LangGraph. Evaluated on 1,516 annotated dialogue records in Vietnamese spanning ten enterprise domains (principally beauty services, professional services, fashion & retail, spa & wellness, and language education, with smaller deployments in marketing, healthcare, banking, and lighting & electrical equipment), GSC-QA achieves a Task Success Rate (TSR) of 87.1% and a Goal Consistency Score (GCS) of 0.78 on the combined 1,516-record evaluation set, and an Error Recovery Rate (ERR) of 79.5% on 205 organic pivot transitions, substantially outperforming standard RAG (58.7% TSR, GCS 0.62). All baselines including Instruct-Only are evaluated on the same dataset for fair comparison. Cross-domain evaluation across 980 records from 14 enterprise deployments shows the mechanism transfers with little loss: on a curated 299-record subset scored under the same judge and rubric as the primary set, task success reaches 62.2% against 63.2% in-domain. Against four external baselines, including an agentic ReAct system, a memory-augmented long-context model, a SimpleTOD-style tracker and a graph-structured stateful agent, GSC-QA leads on task success, faithfulness and answer relevancy alike, and its lead in task success holds under four independent judge families. An LLM-as-judge metric (gpt-5.4) achieves 94% agreement with human annotations. Ablation studies confirm that goal persistence and expert instructions are complementary; removing either component causes significant performance degradation.
A new benchmark, KIFI, is designed, which comprises 1032 carefully selected instances from the TRUE and ScreenEval datasets, with key information annotated, and it is shown that LLMs frequently fail to use the appropriate information to make correct decisions.
Xindi Guo, Zhen Xie, Patrick H. Chen· Annual International ACM SIG...· 0 citations
Collaborative dialogue can end with apparent agreement while participants still differ on goals, assumptions, or execution plans, creating an \textbf{illusion of alignment (IoA)}. A real-user study across 18 meetings confirms that IoA arises routinely in human collaboration. Yet IoA poses a paradox: if participants were aware of such disagreements, they would already be explicit; if not, they cannot articulate them when asked, leaving IoA invisible to both participants and observers. In this work, we make IoA detectable by generating diagnostic multiple-choice questions whose divergent answers across participants provide direct behavioral evidence of hidden disagreement. We construct \textbf{IoA-Suite}, a dataset and evaluation protocol for detecting hidden disagreement, spanning five task types and six domains. We find that even the best model attains only 49.5\% F1, with the bottleneck traced to private context that the dialogue does not surface. We then train \textbf{IoA-Prober-8B} based on IoA-Suite, reaching 51.8\% F1 on IoA-Suite. Across the aforementioned 18 real meetings, it surfaces 2.89 hidden disagreements per meeting that participants confirm they had not voiced, transferring to live human dialogue. Further, in multi-agent collaboration, pairing IoA-Prober-8B with LLM agents improves downstream task performance on BigCodeBench-Hard and HiddenBench.
Kai Liu, Fuwen Luo, Ziyue Wang et al.· 0 citations
Large Language Models (LLMs) excel at tasks like classification, summarisation, question answering among others, with performance comparable to humans. Despite these capabilities, leveraging LLMs to transform unsafe responses in context-sensitive dialogues is underexplored. In this work, we propose a pipeline that leverage LLMs as safety detector, editor and evaluator to mitigate undesired behaviour in human-computer dialogues. At the first iteration, our experimental results on two evaluation datasets show reduction in the unsafe dialogues from 47% to 13% and 48% to 2% respectively, with 82% and 92% agreement between the safety detector and evaluator after revision. Human evaluation of randomly sampled dialogues demonstrates reduction in unsafe responses after revision. Additionally, the revision LLM (editor) exhibits a higher proportion of refusals without compromising fluency and coherence of the revised dialogues.
T. Ajayi, M. Arcan, P. Buitelaar· WOCHAT2026: Workshop on Chat...· 0 citations
Retrieval-based dialogue systems aim to select a proper response according to multi-turn conversational history. Persona-based conversation utilizes prior knowledge to maintain persona consistency, enhancing retrieval accuracy. However, reference-based evaluation relies on high-quality data annotation, which is costly and time-consuming. To address this, we discover persona-centric metamorphic relations to infer test samples from annotated data, without additional annotation cost. Benefiting from this, this work efficiently evaluates the robustness of personalized dialogue models regarding persona consistency. Specifically, we discover three types of metamorphic relations from three aspects: self-persona, partner-persona, and response, to automatically derive new test samples . Then the inherent inference relations between originals and derivatives allow for robustness evaluation. Using this evaluation methodology, our work assesses three widely used training paradigms: non-pretraining, fine-tuning after pre-training, and prompt learning, in personalized dialogue retrieval to observe whether these paradigms are more robust or exhibit the same flaws as the other two paradigms. Our experimental results, based on the three discovered metamorphic relations with consistent outputs reveal that prompt learning is more robust than training from scratch and fine-tuning. While traditional reference-based validation and natural language processing methods achieve competitively high retrieval accuracy (Hits@1 up to 87.4%), the persona consistency of dialogue retrieval systems is just 20.98% when persona descriptions are perturbed using various metamorphic relation-based transformations.
Lin Li, Xiaohua Wu, Yanbing Chen et al.· Cognitive Computation· 0 citations
Human-like private chat requires more than fluent response generation: a system must preserve persona, relationship, memory, bounded knowledge, medium-specific timing, and a coherent multi-turn arc. We present AnthroDial, a closed-loop framework that formulates anthropomorphic dialogue as a joint problem of system architecture, executable evaluation, and diagnostic alignment. It combines (1) a role-conditioned scheduled dialogue runtime with persona and scenario cards, long-term memory, virtual time, and single-draft message decisions; (2) an executable benchmark with an L0 validity gate, five per-turn dimensions, and five dialogue-level dimensions; and (3) a post-training pipeline that filters 16,436 scheduled-decision examples for SFT and applies GRPO with a cognitive-diagnostic, ZPD-aware reward. The reward maintains Kalman-filtered capability estimates for each behavioral dimension, upweights dimensions with larger capability deficits, and uses rollout scores as task-level ZPD matches to focus optimization on learnable weak skills. On a benchmark with 55 personas, 50 scenarios, 50 persona-scenario bindings, and 100 role-conditioned cases per model, we evaluate 16 systems spanning frontier baselines, open models, thinking/no-think variants, and SFT/RL ablations. The strongest non-trained baseline reaches 32.00% strict ACC, while Qwen3.6-27B-SFT+RL reaches 39.00% strict ACC and a 98.5 overall score. In the 9B no-think setting, SFT and RL improve strict ACC from 0.00% to 13.00% and 18.37%. These results show that anthropomorphic dialogue benefits when generation, evaluation, and reward shaping share the same behavioral dimensions.
Wentao Liu, Siyu Song, Xi Chen et al.· 0 citations