From Single Modality to Multimodality in Dysarthric Speech Recognition: A Systematic Scoping Review

Abstract

Dysarthric speech recognition presents significant challenges due to the diverse nature of speech impairments and limitations in available data. Although conventional audio-only automatic speech recognition (ASR) systems have advanced significantly for typical speech, they often exhibit high error rates when faced with the atypical prosody, slower speaking rates, and imprecise articulation that characterize dysarthria. This systematic scoping review investigates the transition from single-modality (audio-only) methods to multimodal approaches by examining how the integration of additional data streams—such as visual cues (e.g., lip movements, facial expressions) or articulatory data (e.g., obtained via electromagnetic articulography [EMA])—may influence the performance and robustness of dysarthric speech recognition systems. We begin by reviewing audio-only approaches based on input feature types, examining how different acoustic representations affect performance on dysarthric speech. We then explore the transition to multimodal systems, discussing fusion strategies and synthesizing audio-visual and audio-articulatory methods within their study-specific evaluation protocols. The survey further addresses open challenges, including data scarcity, speaker variability, and sensor-related complexities, and outlines future research directions, focusing on robust data augmentation, transfer learning, and dynamic fusion architectures. By systematically mapping the progression from single-modality solutions to increasingly sophisticated multimodal frameworks, this paper aims to guide researchers toward more effective and inclusive assistive speech technologies for individuals with dysarthria.