When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition

Mechanistic Insights into Context Failures of Audio-Language Models for Impaired Speech

Analyzing Speech Condition Effects in Dysarthric ASR: A Layer-wise Probing Study

Automatic speech recognition (ASR) performance degrades sharply on dysarthric speech, yet how disordered articulation reshapes a model's internal representations is underexplored. We conduct a layer-wise probing analysis of a transformer ASR encoder on Mandarin dysarthric speech under three transcript-matched conditions: original dysarthric speech, speaker conditioned zero-shot TTS resynthesis, and unconditioned TTS. Probing reveals a task- and condition-dependent representation hierarchy: phoneme boundary information remains weak across all layers for dysarthric speech; phoneme identity is recoverable in deep layers for synthetic speech, but remains poor for dysarthric speech; and recognition difficulty is concentrated in the deepest layers. Furthermore, lexical tone is a persistent error source across all conditions. Guided by these insights, layer-selective LoRA shows that mid-layer adaptation (layer 7 or layers 5-8) recovers near-full encoder performance on dysarthric speech within 6.67% and 2.89% relative margins while training only 0.16% and 0.65% of adapter parameters. Conversely, upper-layer adaptation benefits synthetic speech more than dysarthric speech. These findings link representation analysis to parameter-efficient fine-tuning and motivate layer-aware adaptation for low-resource Mandarin dysarthric ASR.

Leveraging automatic forced alignment of reading passages in dysarthria

Manual acoustic-phonetic segmentation of dysarthric speech is challenging due to the degraded nature of the acoustic signal. The Montreal Forced Aligner (MFA) is a gold-standard approach for automatic segmentation in non-disordered speech, but its efficacy for speakers with dysarthria has not been evaluated. We evaluate the MFA’s performance on vowel segmentation in a standardized reading passage produced by five talkers with dysarthria and five healthy controls. MFA input includes speech audio and an orthographic transcript and uses pre-trained acoustic models and pronunciation dictionaries to time-align word and phone boundaries. We manipulate three alignment conditions that differ in the amount of pre-processing a researcher may choose to perform on their input transcripts, namely (1) the full text of the reading passage, (2) the passage segmented into utterances, and (3) the passage segmented into utterances, with speech errors corrected in the text. We compare the force aligned output with manually segmented vowels as a function of condition, speaker group, and vowel. Alignment across all conditions was high ( > 90% accuracy) for controls, but varied widely for dysarthric speech. Alignment accuracy for dysarthric speakers ranged from 20% (full-text) to 76.4% (segmented-corrected-text). Findings will guide best-practices for leveraging automatic acoustic tools in disordered speech research.

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

Cognitive impairment (CI) is a growing public health concern. Early and accurate diagnosis is critical for enabling timely intervention and improving patient outcomes. Speech-based CI detection has emerged as a promising non-invasive approach, as speech signals encode both linguistic and acoustic markers associated with cognitive decline. Recent advances in large language models (LLMs) further strengthen the potential of speech-based assessment by enabling more expressive representation learning and improved generalization across diverse speakers, recording devices, and clinical environments. Moreover, multimodal learning by jointly modeling linguistic and acoustic features allows for a more comprehensive characterization of cognitive and behavioral changes related to CI, leading to more reliable detection. In this work, we propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts while preserving patient privacy. Acoustic embeddings are extracted directly from speech signals, while textual embeddings are generated from automatically transcribed speech. These modality-specific embeddings are then concatenated to create a combined feature vector and used for downstream classification, without requiring access to raw or sensitive patient data. The proposed approach is evaluated on the ADReSS20 and ADReSSo21 benchmark datasets. Experimental results show that the proposed multimodal framework achieves an CI classification accuracy of 92.4% and consistently outperforms single-modality baselines. Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization. This advance highlights the power of an LLM-based multimodal framework that fuses linguistic and acoustic data to enable robust, scalable, and non-invasive screening.

Model Development and Validation for Repetition Severity Assessment in Stuttered Speech Using Clinical Speech Datasets

Stuttering disrupts the forward flow of speech through involuntary repetitions, prolongations, and blocks, with repetitions being the most common and clinically telling of the three. Quantifying how severe those repetitions are matters for diagnosis, therapy planning, and tracking whether treatment is actually working. The catch is that severity has long been judged by ear, by trained speech-language pathologists counting disfluent events and assigning ratings on standardised scales. That process is slow, variable between clinicians, and constrained by who is available. This paper builds and tests a computational model that grades repetition severity from clinical speech recordings. The model draws on a multimodal feature set combining acoustic, prosodic, temporal, and spectral descriptors, evaluated on 480 audio samples from 60 adult speakers with persistent developmental stuttering. Two certified clinicians labelled each sample as mild, moderate, or severe, with strong inter-rater agreement. Recursive feature elimination trimmed the feature set to a compact, discriminative subset, and five classifiers were trained under stratified ten-fold cross-validation repeated five times. A Gradient Boosted Trees model reached 91.46 percent mean accuracy, a macro F1-score of 0.90, and a Cohen kappa of 0.86, ahead of Random Forest, a Support Vector Machine, a Multilayer Perceptron, and Logistic Regression. Per-class scores were strong for mild and severe cases and weaker for moderate, where acoustic characteristics overlap with both neighbouring classes. Friedman and Nemenyi tests confirmed the top model's lead was significant at the 0.05 level. The pipeline is reproducible and the results support its use as a clinical decision-support tool.

LoRA-MoE Fine-Tuning for Improved Speech Recognition in People With Parkinson’s Disease

Despite advances in automatic speech recognition (ASR) technology, its performance remains limited when processing dysarthric speech, which hinders people with dysarthria from accessing voice-based digital technologies. In this study, we propose LoRA-MoE, a parameter-efficient adaptation method that combines Low-Rank Adaptation (LoRA) with a mixture of experts (MoE), to improve speech recognition for individuals with Parkinson’s disease (PD), who often exhibit hypokinetic dysarthria. We integrate LoRA-MoE into the Whisper model by freezing its pre-trained weights while introducing multiple LoRA experts and a router network that dynamically combines them based on the input. Experiments on a Korean PD speech dataset show that LoRA-MoE achieves a character error rate (CER) of 12.01%, a 38.54% relative improvement over the baseline Whisper model, and outperforms full fine-tuning and single LoRA adaptation by 46.00% and 11.37%, respectively, with consistent improvements across all PD severity levels. The proposed approach demonstrates consistent improvements and stable performance across all severity levels, and its performance is robust to the number of experts; we adopt three experts as our default configuration.