Skip to content

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

Sep 2026 · arXiv.org · Vol abs/2609.38501 · 0 citations · 29 references
Engineering Computer Science

TL;DR

The proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning, and are presented as a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution.

Abstract

Multi-user voice agents must track who said what across dialogue sessions. Text LLMs are attractive backbones for such agents, but transcripts alone do not expose acoustic speaker identity, leaving the model without a persistent reference for linking information to speakers across sessions. We address this gap by introducing Speaker Handles, soft-token representations that expose acoustic speaker identity to a frozen text LLM for cross-session speaker-dependent reasoning. A three-stage curriculum trains a lightweight projector, with fewer than 0.1% of the backbone's parameters, to map speaker embeddings into these handles. Establishing whether the resulting handles truly support cross-session speaker-dependent reasoning is challenging with existing benchmarks because textual cues can partially reveal fact ownership. We therefore present SpeakerBind, a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution. Speaker Handles achieve 97.40-98.36% accuracy on VoxCeleb1 and 70.40% on SpeakerBind, close to the 71.88% topline. These results show that the proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents

Personalized spoken agents must identify speakers before accessing user-specific memory, yet memory benchmarks typically assume correctly attributed histories. We argue that voiceprints are observations of identity, not identities themselves, and formulate online speaker-memory construction to maintain persistent user...

Yu-Cong Zhang, Can-Can Li, Bo Xiao et al. · 0 citations

Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings

Speech transcripts used as long-term memory must preserve both words and stable speaker identities. Existing meeting-transcription metrics either ignore speakers or remap anonymous speakers independently in each recording, so they cannot measure whether the same person retains one identity across meetings. We evaluate...

S. Vispute, A. M. Mishra, Siddhartha Saxena · 0 citations
Preprint Oct 2026

SpeakerTurn: Target-Speaker Semantic Turn Detection in Multi-Speaker Environments

Semantic completeness alone cannot determine when an assistant should respond in a multi-speaker environment: a completed utterance may belong to an interfering speaker, while the designated user is still speaking, acknowledging, or requesting time. We therefore study semantic turn inference conditioned on speaker iden...

Nan Li, Hetao Shan, Jin-Tao Liang et al. · 0 citations

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

SpeakerMem-R1 is proposed: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time, and ablations show that the verbatim and structured tracks, as well as person-...

Hao-Bo Zheng, Tan Tang, Yan Chen et al. · 0 citations
Preprint Oct 2026

Toward accurate speaker inventories for online speaker diarization

An online speaker diarizer maintains a growing inventory of speakers, and applications read it directly: a live transcript enumerates them as they appear. Existing systems either fix the inventory's size in advance, so speakers beyond that size are merged, or grow it on distance alone, so noise and turn boundaries spli...

Young-ki Kwon, Hee-Soo Heo, Min-Ji Lee et al. · 0 citations
#artificial intelligence Preprint Oct 2026

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as"When did I originally plan to leave?"or adapt their behavior to individual users based on past interactions. This requires more than retrieving a topically similar passage: the assistant mu...

Achira Lin, Si-Yuan Hou, Wen-Yi Yu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.