Jul 2026· SIGDIAL Conferences· pp. 614-635· 0 citations· 26 references
Computer Science
TL;DR
TRACER detects useful failure signals well before the end of the conversation and outperforms heuristic, classical, and single-stream baselines, suggesting that early failure detection can provide a practical warning signal for dialogue systems before the interaction fully breaks down.
Abstract
Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has already gone wrong. We present TRACER, a method for early failure detection in task-oriented dialogue. TRACER predicts from a partial dialogue whether the full conversation will eventually fail by combining simple trajectory signals from belief-state changes with text representations of the evolving dialogue state. We evaluate the method in both oracle and generated belief-state settings, and test how well it works when only 25%, 50%, 75%, or 100% of the dialogue is visible. Across these settings, TRACER detects useful failure signals well before the end of the conversation and outperforms heuristic, classical, and single-stream baselines. These results suggest that early failure detection can provide a practical warning signal for dialogue systems before the interaction fully breaks down.
Dialogue state tracking (DST) is one of the core components in task-oriented dialogue systems. At each turn in a conversation, DST estimates the user belief or dialogue state, which is used as input for downstream modules to predict system actions and generate responses. The increasingly popular dialogue system applications like Google Assistant, Siri and Alexa need to support a large number of services and APIs, resulting in growing attention to the scalability of such systems. Especially for some domains with little or no training data, the capability of transferring existing knowledge of other domains is highly desired. In this paper, we present a novel scalable framework for multi-domain dialogue state tracking. The proposed system leverages the pretrained BERT model to achieve zero-shot generalization, making it easy to quickly adapt to new domains without additional training. The performance of our model is evaluated on recently released schema-based dialogue (SGD) dataset, showing significant improvement compared to previous baseline.
Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.
An isolated final user message is often treated as the query in evaluations of AI systems. In a conversation, however, the actionable request may be distributed across preceding turns. We directly test whether that omitted within-conversation context changes answers. For each of 180 English multi-turn conversations sampled from a governed commercial corpus and the public PRISM dataset, we hold the final user message and requested answer model constant while generating three answers: one from the full role-labelled conversation, one from the final message alone, and one from the final message plus a prefix-only reconstruction capped at 160 words. A separately requested judge model evaluates answers under randomized labels. The prespecified primary endpoint is a material difference that could change what the user does, rather than a difference in style or detail. After inverse-probability weighting to the eligible cohorts, the full-conversation and isolated-final answers differ materially in 44.7% of cases (95% bootstrap CI 33.8% to 56.1%). Full-conversation answers score 0.49 points higher on a 0 to 4 request-satisfaction scale (0.32 to 0.67). Adding the compressed prefix reduces the material-difference rate to 30.8% (20.2% to 42.1%), a 13.9-point reduction (4.9% to 24.1%), and reduces the mean satisfaction gap to 0.01 points (-0.12 to 0.13). Yet compression is not equivalent to the complete dialogue context: almost one third of answers remain materially different. An order-swapped repeat on 48 cases yields 91.7% agreement and kappa = 0.83 for the primary decision. The study concerns preceding turns in the same conversation and does not test persistent memory across separate conversations.
Classical spoken dialogue systems often separated dialogue management from response realization: a policy selected the next dialogue action, and a generation component expressed that action. As dialogue systems shift toward LLMs, this decomposition has largely disappeared into the model's hidden representations. We ask whether an LLM-internal analogue of state estimation and action control can be recovered for conversational moves such as acknowledging, checking, querying, explaining, and replying. We formulate move control as two coupled problems: selection, predicting the appropriate next move from the dialogue context, and realization, causally producing a chosen move at generation time. We introduce Latent-IM, an internal dialogue-management framework that provides a general interface for choosing and deploying conversational moves under different objectives. Here, we use this control to reproduce human move choices, improving average end-to-end move accuracy by 12.5 points over the unsteered backbone while performing comparably to fine-tuning.
Adar Avsian, Atahan Dokme, Tony Woo et al.· 0 citations
This work investigates a method for automatically detecting inconsistencies in Task-Oriented Dialogues by conceptualizing TODs as a Constraint Satisfaction Problem (CSP), where variables represent dialogue segments referencing the conversational domain and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge.
Tiziano Labruna, Giovanni Bonetta, Bernardo Magnini· 0 citations
Traditional conversational Question Answering (QA) systems focus on isolated information retrieval and lack the ability to maintain goal persistence across multi-turn interactions. Current Retrieval-Augmented Generation (RAG) approaches operate passively, retrieving information turn-by-turn without tracking whether the overall task objective has been achieved, and are typically evaluated using linguistic similarity metrics rather than functional task outcomes. We introduce GSC-QA (Goal-based Sequential Conversation QA), a framework that integrates three complementary components into a unified enterprise dialogue architecture: 1) persistent Goal Tracking Module (GTM) that maintains task state across turns in external databases (Redis + PostgreSQL), decoupled from the LLM context window; 2) Expert Instruction Layer that injects domain-curated procedural constraints into every response generation step; and 3) proactive slot detection that identifies and elicits missing required information before proceeding. Unlike prior task-oriented systems that assume static ontologies or session-scoped memory, GSC-QA combines retrieval, instruction enforcement, and goal persistence in a single coordinated loop built on LangGraph. Evaluated on 1,516 annotated dialogue records in Vietnamese spanning ten enterprise domains (principally beauty services, professional services, fashion & retail, spa & wellness, and language education, with smaller deployments in marketing, healthcare, banking, and lighting & electrical equipment), GSC-QA achieves a Task Success Rate (TSR) of 87.1% and a Goal Consistency Score (GCS) of 0.78 on the combined 1,516-record evaluation set, and an Error Recovery Rate (ERR) of 79.5% on 205 organic pivot transitions, substantially outperforming standard RAG (58.7% TSR, GCS 0.62). All baselines including Instruct-Only are evaluated on the same dataset for fair comparison. Cross-domain evaluation across 980 records from 14 enterprise deployments shows the mechanism transfers with little loss: on a curated 299-record subset scored under the same judge and rubric as the primary set, task success reaches 62.2% against 63.2% in-domain. Against four external baselines, including an agentic ReAct system, a memory-augmented long-context model, a SimpleTOD-style tracker and a graph-structured stateful agent, GSC-QA leads on task success, faithfulness and answer relevancy alike, and its lead in task success holds under four independent judge families. An LLM-as-judge metric (gpt-5.4) achieves 94% agreement with human annotations. Ablation studies confirm that goal persistence and expert instructions are complementary; removing either component causes significant performance degradation.