Skip to content

Distributional Metrics for Evaluating Spoken Conversational Systems

Sep 2026 · arXiv.org · Vol abs/2609.31719 · 0 citations · 34 references
Computer Science Engineering

TL;DR

The Conversational Distribution Score (CDS) is introduced, which compares distributions of conversational behaviour using human conversations as a reference, and how many minutes and conversations are required before rankings stabilize is examined.

Abstract

Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal--oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.

View source

Similar papers

Book Open access Sep 2026

Conversational Style in Open Domain Dialogue Systems: What Makes a Response Sound Natural

Conversational style in this setting is best understood as a pragmatic orientation toward the user rather than toward the system’s own content, and that the system must be mixed-initiative to manifest a conversational style.

Vrindavan Harrison, M. Walker · 0 citations
Conference Open access Sep 2026

From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models

A French full-duplex benchmark with two variants, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, is introduced and a trade-off between optimizing benchmark metrics and preserving conversational naturalness is found.

Hamid Soltani, Gilles Boulianne · 1 citation
#artificial intelligence Preprint Sep 2026

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

ReACT-TTS is proposed, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization, and connects the predicted response style to a Grad-TTS backbone for end-to-end speech realization.

Yun-Ji Chu · 0 citations
#natural language process... Preprint Sep 2026

Same Words, Different Actions: Paired Turn-Taking Evaluation under Rewritten Dialogue Contexts

Real-time spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Existing benchmarks typically score events independently and may therefore assign high scores to systems with fixed action preferences rather than context-sensitive decision...

Shuo-Feng Zhao, Hong-Wei Cai, Wen-Ke Fan et al. · 2 citations
#natural language process... Conference Open access Sep 2026

BaatCheet: A Multilingual Corpus for Dialogue Translation in Indian Languages

BatCheet is introduced, a multilingual dialogue corpus named after the Hindi term for conversation or chitchat, containing approximately 49,000 dialogues for dialogue translation across five translation directions and fine-tuning five open-source LLMs yields substantial gains over zero- and few-shot baselines.

Priyanka Dasari, Yuvrajsinh Bodana, Vandan Mujadia et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.