Skip to content
Review

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

Sep 2026 · arXiv.org · Vol abs/2609.35613 · 0 citations · 217 references
Computer Science

TL;DR

A review of the current landscape and open problems in multimodal TSE from the perspective of auxiliary target cues drawn from multiple modalities and summarizes future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment.

Abstract

Target Speaker Extraction (TSE) is pivotal in speech communication and human-computer interaction, enabling the isolation of a specific speaker's voice from complex acoustic environments, i.e., the cocktail party scenario. Although traditional TSE systems conditioned on enrollment speech have progressed substantially, enrollment speech as a cue has inherent limitations. Its reliability degrades when the target and interfering speakers have similar voice characteristics, when intra-speaker variability (e.g. changes in emotion or speaking style) creates a mismatch between the enrollment and target speech, or when the enrollment itself is contaminated by noise or competing speakers. This review surveys deep-learning-based TSE from the perspective of auxiliary target cues drawn from multiple modalities. We organize existing methods according to five types of information used to isolate the target speaker: audio enrollment, visual, spatial, textual/semantic, and neural cues. We also trace the evolution from discriminative estimators to variational, diffusion, flow, codec, and foundation-model-based systems and summarize representative datasets and evaluation metrics. We review the benefits and limitations of different cues and discuss challenges involving synchronization, missing or unreliable observations, data scarcity, privacy, computational cost, and real-time operation. Finally, we summarize future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment. By jointly reviewing cue design, model architecture, training objectives, datasets, and evaluation metrics, this article provides an overview of the current landscape and open problems in multimodal TSE.

View source

Similar papers

Preprint Sep 2026

Unified Target-Speaker ASR with Text and Enrollment Speech Cues

Target-speaker automatic speech recognition (TS-ASR) aims to recognize a designated speaker while suppressing interfering speech in multi-talker environments. Conventional TS-ASR typically relies on an enrollment utterance, whereas text-guided methods use known lexical content, such as a wake word, to identify the targ...

Yu-Xiang Mei, Yu-Chen Yan, Dong-Xing Xu et al. · 0 citations
Preprint Oct 2026

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

Generative TSE model PromptSep-Speaker is proposed, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption that achieves state-of-the-art performance over TSE and universal source separation baselines, improving...

Yu-Tong Wen, Ke Chen, Prem Seetharaman et al. · 0 citations
Preprint Sep 2026

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

This work proposes an LLM-based target-speaker voice activity projection (TS-VAP) module, which forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-la...

Shu-Han Zhang, Wen-Xuan Wu, Hai-Zhou Li · 0 citations
Preprint Sep 2026

Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction

Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized...

Rayhan Rashed, Senja Filipi, Ross Cutler · 0 citations
Open access 2026

Conditional Speaker Normalization of Vocal Tract Shape Features for Robust Synthetic Speech Detection

A Conditional Speaker Normalization module is introduced that conditions the normalization process on speaker embeddings derived from either speaker verification systems or a speaker-height estimation model to mitigate the speaker-dependent nature of Log-Area Ratios.

Hossein Fayyazi, Yasser Shekofteh · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.