Skip to content
Preprint

Identity-Faithful Audio-Visual Target Speaker Extraction with REAL-2MIX and VOXBLINK2-AVSE

Aug 2026 · 0 citations · 30 references
Engineering

TL;DR

REAL-2MIX, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video, and REAL-2MIX-AVSE, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video are introduced.

Abstract

Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video. Each scene also contains preceding A-only and B-only stages that provide in-scene speaker references. It contains 77 scenes and 7,598 clips (11.84 hours), including 6,042 dual-annotated mixtures. After processing there leave 6,038 evaluable mixtures and 12,076 target-speaker rows. We additionally curate VOXBLINK2-AVSE from VoxBlink2, comprising 250,828 synchronized audio--lip-ROI pairs from 28,421 identities and 766.17 hours of speech. Our extractor uses frozen, 1,280-dimensional projected AV-HuBERT features, target-conditioned training, and layer-wise feature modulation. We jointly evaluate content with Qwen3-ASR-1.7B CER and target identity with WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD). On the complete manifest, the best archived checkpoint obtains 0.2261 CER, 82.22% strict output correctness, and 69.53% both-output strict success.

View source

Similar papers

Preprint Aug 2026

The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge: A Benchmark with Natural Mixtures and Degraded Video

Audio-visual speech enhancement (AVSE) uses a target speaker's visible articulation to recover that speaker's voice from overlapped speech. Yet most evaluation protocols rely on synthetic mixtures and reliable video. The ISCSLP 2026 Real-World AVSE Challenge addresses both gaps. Track~1 combines naturally recorded two-...

Kai Li, Wen-Ze Ren, Jun-Jie Li et al. · 0 citations
Preprint Sep 2026

Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction

Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized...

Rayhan Rashed, Senja Filipi, Ross Cutler · 0 citations
Preprint Sep 2026

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

This work proposes an LLM-based target-speaker voice activity projection (TS-VAP) module, which forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-la...

Shu-Han Zhang, Wen-Xuan Wu, Hai-Zhou Li · 0 citations
Review Sep 2026

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

A review of the current landscape and open problems in multimodal TSE from the perspective of auxiliary target cues drawn from multiple modalities and summarizes future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment.

Xin-Yuan Qian, Yang Zhou, Zi-Yang Jiang et al. · 0 citations
Preprint Oct 2026

BabelFake: A Multilingual Audio-Visual DeepFake Benchmark

B BabelFake is introduced, a multilingual audio-visual DeepFake benchmark recorded with consenting participants, and it is shown that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved.

Carlotta Segna, Joel Tschesche, Anna Rohrbach · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.