Modern multimodal foundation models (MFMs) have made rapid progress on tasks requiring integrated perception across speech, vision, and language, including emotion recognition. However, it remains unclear whether they recognize speech and facial emotion through shared affective functional units or modality-specific pat...
Xiutian Zhao, Lu Sun, Björn W. Schuller et al.· 0 citations
The loss of speech limits communication for individuals with paralysis. Direct neural-to-speech synthesis is challenging due to the limited availability of neural data for training speech brain-computer interfaces. Most existing systems rely on cascaded neural-to-text-to-speech pipelines, which increase inference laten...
A novel unsupervised domain adaptation method, Iterative Adversarial Self-Training (IAST), is proposed and results demonstrate that IAST significantly improves the generalization ability and robustness under various cross-domain settings.
Lu Sun, Shreeram Suresh Chandra, Aurosweta Mahapatra et al.· 0 citations
This work introduces AffectDF, the most comprehensive benchmark for emotionally expressive speech deepfakes, spanning TTS, VC, emotional VC, and LALM-based spoofing attacks across both acted and spontaneous emotional speech.
Zero-shot text-to-speech (TTS) can clone a speaker's voice from a short audio prompt, yet most TTS systems still require the audio prompt transcript during inference. This dependency prevents cross-lingual voice cloning when the audio prompt transcript is unavailable, particularly for unseen languages. Cross-Lingual F5...
Qingyun Liu, Rixi Xu, Yu-Shen Chen et al.· 0 citations
A replay-regularized, attack-aware curriculum that steps exposure based on measured attack influence is proposed that shows improved overall robustness and reduced attack-level imbalance compared with standard multi-attack training.
Avantika Singh, Aurosweta Mahapatra, I. R. Ulgen et al.· 0 citations
This work proposes ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency.
Yu-Pei Li, Qi-Yang Sun, Mohamed Mady et al.· 0 citations
These findings provide the first causal, neuron-level account of how LALMs encode emotion across languages, and establish multilingual neuron identification as an effective mechanism for understanding cross-lingual affective behavior.
Xiutian Zhao, Philipp Koehn, Björn W. Schuller et al.· 1 citation
Brain2Speech-Net is presented, among the first single-stage frameworks to remain intelligible under limited data while removing intermediate text decoding, and achieves strong intelligibility in objective and listening tests while running faster than real time.
The role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection is revisited, suggesting that ``cleaner''speech datasets are not necessarily more reliable for real-world AD detection.
Lu Sun, Shreeram Suresh Chandra, Lin Zhang et al.· 0 citations
This work presents the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs and reveals inherent cross-lingual correlations in prosodic structure between certain languages.
Haopeng Xie, I. R. Ulgen, Sofia Son et al.· 1 citation
DiffAnon is proposed, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation, and is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control.
Ismail Rasim Ulgen, Zexin Cai, Nicholas Andrews et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
