This work develops ParA-LLM, a framework of 22 paralinguistic characteristics that surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench and releases ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories.
Abstract
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
CASA, a simpler architecture combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art performance while providing a more interpretable separation between speech delivery and content, is proposed.
Nhan Phan, Ilona Lähteenmäki, Anna von Zansen et al.· 0 citations
Candor-LR is introduced, a conversational benchmark derived from the CANDOR corpus of 1,656 natural dyadic videoconferences that significantly improves cross-domain robustness under both clean and noisy conditions, as its realistic conversational data captures broader audio-video features.
Rishabh Jain, A. Papadopoulos, Zhao-Feng Lin et al.· 0 citations
Mizar, a 159.3M-parameter ALM, is introduced, a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM that surpasses the previous best-performing ALM below 200M parameters on all three benchmarks.
FireRedAudio is introduced, a general-purpose audio language model with a shared 9B-parameter LLM that achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantia...
Fei-Yu Shen, Fenglong Xie, Junjie Li et al.· 3 citations· ⚡1
This work proposes a framework that fine-tunes LALMs for dysarthric speech detection on speech recordings combined with textual information comprising low-level acoustic features and speaker demographics, and establishes an effective approach for adapting LALMs to dysarthric speech detection.
Mahdi Amiri, H. Shahreza, Pascal Frossard et al.· 0 citations
Qwen-Audio-3.0-ASR is presented, a Mixture-of-Experts (MoE) LLM-based ASR system designed to address production demands through a unified, instruction-following framework and demonstrates state-of-the-art or highly competitive recognition performance across a broad range of evaluation conditions.
Chuan-Meng Bian, Da-Ren Chen, Pei-Xin Chen et al.· 3 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.