IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information, improves average accuracy and improves a model already explicitly fine-tuned to prioritize acoustic evidence.
Abstract
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.
Research on automatic speaking assessment (ASA) has increasingly adopted multimodal speech large language models to assess learners'speaking performance. However, existing studies provide limited analysis of how acoustic and content information contribute to predictions and how stable the resulting performance is. We propose CASA, a simpler architecture combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art performance while providing a more interpretable separation between speech delivery and content. On the Speak&Improve Corpus 2025, CASA achieves a root mean square error (RMSE) of 0.358, improving on the previous best RMSE while using approximately half the estimated inference parameters. The general-purpose architecture is designed for adaptation to other ASA corpora without structural changes and relies on three handcrafted fluency features. Through ablations and repeated runs, we analyze the individual and complementary contributions of acoustic and content information, examine performance variability, and demonstrate the potential of large language model reasoning for training-free content validation.
Nhan Phan, Ilona Lähteenmäki, Anna von Zansen et al.· 0 citations
Experimental results show that VALL-E outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity and could preserve the speaker’s emotion and acoustic environment from the prompt in synthesis.
This work trains an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models.
Harsha Vardhan Khurdula, Abhinav Singh, Yoeven D. Khemlani et al.· 0 citations
Conventional techniques for speech and music discrimination often rely on handcrafted features and shallow models, leading to limitations in capturing complex temporal dependencies and handling noisy or degraded audio signals. To address these challenges, this paper proposes a novel approach that employs attention-based deep neural networks (DNNs) for improved speech and music classification. The proposed method utilizes a combination of LSTM and attention mechanisms to effectively model the temporal dynamics of audio signals and focus on the most discriminative parts. By incorporating attention, the model can learn to weigh different segments of the audio signal based on their relevance for classification. Experiments are conducted on two publicly available standard speech/music datasets: SS and GTZAN, to evaluate the performance of the proposed method. Initially, we have found that a combination of Mel-frequency cepstral coefficients (MFCCs), spectral roll-off deviation, spectral flux deviation, and zero-crossing rate (ZCR) deviation provides the best performance for speech and music discrimination using a Gaussian mixture model (GMM) classifier, achieving an accuracy of 92.68%. By incorporating attention mechanisms on the hybrid features (simple, LSTM and Hierarchical), the model can focus on the most discriminative parts of the audio signal, leading to improved classification accuracy (test accuracy of 99.58% for SS and 97.58% for GTZAN) and generalization over the previous work. The results demonstrate significant improvements over conventional techniques in terms of accuracy, precision, recall, and F1-score.
Arvind Kumar, Biswajit Karan, J. Sahoo et al.· 2026 6th International Confe...· 0 citations
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Qingjian Lin, Yuxin Li, Haoyang Zhang et al.· 1 citation
ReLMCodec is a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle that moves the empirical single-stream predictability--reconstruction frontier in the evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similarity.
Zixiang Wan, Xusheng Yang, Zhengmeng Wang et al.· 0 citations