Phoneme-Preserving Mamba-Based Frontend for Noise-Robust Automatic Speech Recognition

Abstract

Speech enhancement models optimised for perceptual quality metrics such as PESQ do not necessarily improve automatic speech recognition (ASR) performance, as aggressive noise suppression can distort the phonemic cues that acoustic models rely on. We propose a time-domain speech frontend based on the Mamba selective state space model, trained exclusively with ASR-oriented loss functions to preserve phonetically discriminative structure rather than maximise perceptual quality. The model follows a U-Net encoderdecoder architecture with 4 × temporal downsampling and six stacked bidirectional Mamba blocks operating on the raw waveform, trained with a multi-component loss combining multiresolution spectral supervision, log-mel feature matching, speech correlation preservation, and ideal-ratio-mask spectral objectives. On VoiceBank-DEMAND, the proposed model achieves 10.26% WER with wav2vec2-base-960h and 7.68% with whisper-base, achieving statistically equivalent WER to SE-Mamba (10.30% / 7.73%) while producing a lower PESQ of 3.10 versus 3.69. Phoneme-class analysis confirms that the proposed model matches or outperforms SEMamba on five of seven articulatory classes, demonstrating that ASR-oriented training objectives achieve equivalent robustness to perceptual enhancement models without modifying the downstream ASR system.