This work evaluates state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias.
Abstract
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.
The results show that LLM- based evaluation can serve as an effective component of large- scale voice-agent assessment, but that its reliability is metric- and configuration-dependent rather than uniform.
Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior and communication, yet validation methods for these applications remain limited. Existing evaluations often rely on human judgments of plausibility, a coarse, difficult-to-scale, and often unreliable benchmark for disti...
Nicolò Pagan, Petter Törnberg, Christopher A. Bail et al.· Proceedings of the National...· 0 citations
The Conversational Distribution Score (CDS) is introduced, which compares distributions of conversational behaviour using human conversations as a reference, and how many minutes and conversations are required before rankings stabilize is examined.
S. Satish, Erica Cooper, Patrícia Schmidtová et al.· arXiv.org· 0 citations
This work introduces OIR annotations for the NOXI corpus, achieving considerable reliability and evaluating whether LLMs can reliably annotate OIR sequences using structured Chain-of-Thought prompting and comparative analysis across two corpora, finding weak alignment between LLMs and human annotations, particularly in...
Anh Ngo, Nicolas Rollet, C. Pelachaud et al.· International Conference on...· 1 citation
JUPE, a visual-question-answering (VQA)-style benchmark that fixes answers while varying explanations is introduced, finding a significant mean shift in decision criterion toward acceptance, without a clear change in mean discriminability.
Ang-Lin Cai, Zhi-Qi Wang, Xiao-Zhi Yang et al.· 0 citations
TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection, finds end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in...
Freeman Jiang, Ramon Sanabria, Soham Deshmukh et al.· 9 citations· ⚡4
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 18, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.