Skip to content

Calibrating LLM Judges for Human and AI Conversations

Sep 2026 · arXiv.org · Vol abs/2609.29431 · 1 citation · 33 references
Computer Science

TL;DR

This work evaluates state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias.

Abstract

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

View source

Similar papers

#data science Open access Oct 2026

AI-generated language remains systematically distinguishable from human communication.

Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior and communication, yet validation methods for these applications remain limited. Existing evaluations often rely on human judgments of plausibility, a coarse, difficult-to-scale, and often unreliable benchmark for disti...

Nicolò Pagan, Petter Törnberg, Christopher A. Bail et al. · 0 citations

Distributional Metrics for Evaluating Spoken Conversational Systems

The Conversational Distribution Score (CDS) is introduced, which compares distributions of conversational behaviour using human conversations as a reference, and how many minutes and conversations are required before rankings stabilize is examined.

S. Satish, Erica Cooper, Patrícia Schmidtová et al. · 0 citations
Open access 2026

Human vs LLM in Conversational Repair Annotation: A New Resource and Comparative Study

This work introduces OIR annotations for the NOXI corpus, achieving considerable reliability and evaluating whether LLMs can reliably annotate OIR sequences using structured Chain-of-Thought prompting and comparative analysis across two corpora, finding weak alignment between LLMs and human annotations, particularly in...

Anh Ngo, Nicolas Rollet, C. Pelachaud et al. · 1 citation
#artificial intelligence Preprint Oct 2026

A Plausible Explanation Is Enough to Mislead Multimodal Judges

JUPE, a visual-question-answering (VQA)-style benchmark that fixes answers while varying explanations is introduced, finding a significant mean shift in decision criterion toward acceptance, without a clear change in mean discriminability.

Ang-Lin Cai, Zhi-Qi Wang, Xiao-Zhi Yang et al. · 0 citations
#natural language process... Preprint Aug 2026

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection, finds end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in...

Freeman Jiang, Ramon Sanabria, Soham Deshmukh et al. · 9 citations · ⚡4

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Putting a spotlight on AI, society, and humanity

Supported by the MIT Human Insight Collaborative, scholars are exploring how AI is impacting the human experience and reshaping higher education.

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.