A new loss function is proposed, which helps mitigate the effect of excess silence in training, improving STOI, frequency-weighted segmental SNR, and frequency-weighted segmental SNR from 4.35 to 5.12.
Abstract
Target-speaker and multi-speaker extraction are techniques for extracting speech from a desired speaker or desired speakers in the presence of other speakers and/or noise. Neural network approaches for this task are often trained and evaluated using simulated datasets, with balanced amounts of target speech and speaker enrolment samples which closely match the target speech. However, in real multi-party conversations, participants are often silent for more time than they are speaking, and their enrolment speech samples can differ substantially from the target speech in the conversation. These factors can impact the training and evaluation of these techniques on recordings of real conversations. This work proposes a new loss function, which helps mitigate the effect of excess silence in training, improving STOI from 0.55 to 0.60, and frequency-weighted segmental SNR from 4.35 to 5.12. Additionally, the impact of the mismatch between the enrolment speech and target speech is explored.
A review of the current landscape and open problems in multimodal TSE from the perspective of auxiliary target cues drawn from multiple modalities and summarizes future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment.
Xin-Yuan Qian, Yang Zhou, Zi-Yang Jiang et al.· 0 citations
This work proposes an LLM-based target-speaker voice activity projection (TS-VAP) module, which forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-la...
A Conditional Speaker Normalization module is introduced that conditions the normalization process on speaker embeddings derived from either speaker verification systems or a speaker-height estimation model to mitigate the speaker-dependent nature of Log-Area Ratios.
We introduce target-speaker unlearning ASR (TSU-ASR) task in a fully end-to-end framework for multi-speaker ASR and diarization. Given a multi-speaker utterance and a set of opt-out speakers who do not wish to have their speech transcribed, the task requires an ASR system to transcribe all speakers except the opt-out o...
VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling, reduces the need for a separate speaker encoder, and improves F0 dynamics and vowel-discrim...
Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah et al.· 0 citations
It is shown that fine-tuning a non-autoregressive architecture, such as ForwardTacotron, achieves speaker-adaptation quality comparable to zero-shot methods, with lower computational complexity but a lower mean opinion score.
Kishor Kayyar Lakshminarayana, Frank Zalkow, Christian Dittmar et al.· Journal on Audio, Speech, an...· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comOct 22, 2026
Flint-interactive gives AI agents a language for creating interactive visualizations. Click a chart, filter data, highlight related content, and explore beyond the view the author originally chose. The post Flint-interactive: A language for creating interactive data visualizations appeared first on Microsoft Research.
A new computational approach reveals how cells communicate to assemble into tissues. This could accelerate stem cell engineering, improve disease models, and inform therapies.
Microsoft Research Blog· microsoft.comOct 15, 2026
Zero-knowledge proofs can verify computing results without rerunning workloads or exposing sensitive data. Magnetar helps bring them into the mainstream with support for standard programming languages, major speed gains, and quantum-resistant security. The post Proof-based general-purpose computing with Magnetar appeared first on Microsoft Research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
