A controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus is presented, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs).
Abstract
Developing Automatic Speech Recognition (ASR) for morphologically rich, low-resource languages such as Assamese is challenging due to insufficient annotated speech data. The pretrained Whisper model performs poorly on Assamese speech recognition tasks. This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus. A hardware-aware optimized training pipeline is implemented for resource-constrained environments, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs). The proposed fine-tuned model significantly outperformed the Zero-shot baseline, yielding Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), and Word Infomation Loss (WIL) of 43.17\%, 13.18\%, 43\%, and 64.81\%, respectively, achieving significant relative improvements of 78.26\%, 93.10\%, 57.0\%, and 35.19\% over the baseline. Semantic evaluation of the fine-tuned model also demonstrates notable improvement over a zero baseline, attaining Bilingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of 30.81 and 0.5262, respectively. Additionally, the predicted hallucination rate and Real-Time Factor (RTF) are substantially improved by 96.70\% and 32.38\%, compared to the zero-shot baseline.
A preliminary study on the adaptation of Whisper for Automatic Speech Recognition in Baniwa, an indigenous Arawakan language spoken in Brazil, Colombia, and Venezuela, demonstrating that multilingual foundation models can be successfully adapted to extremely low-resource indigenous languages.
Leonardo Duart, T. Fonseca, T. Chacon· 0 citations
This study reports the development of an Automatic Speech Recognition (ASR) system in Mizo, a low-resource language. The development included collecting 17.62 hours of speech data, curating it, and fine-tuning the Mizo ASR system with three Whisper multilingual models and with the SraVaani 1.0 Indic multilingual model. Whisper-large-v3 achieved the lowest conventional WER (18.08%), while morphology-aware evaluation yielded a WER of 7.22%. Zero-shot evaluation of the SraVaani 1.0 Indic multilingual model yielded a WER of 58.27%, while Mizo-specific fine-tuning reduced the conventional WER to 29.45% and the morphology-aware WER to 17.93%. The results demonstrate that the Whisper model can achieve a substantially low WER, even when adapted to an unseen language. In contrast, SraVaani 1.0 supports the Mizo language in its multilingual model; however, fine-tuning with carefully curated Mizo speech data substantially improves its performance.
Priyankoo Sarmah Sanasam, Ranbir Singh, ID Lalhmingmawia· 0 citations
To address the low accuracy and poor adaptation of generic speech recognition models to unique pronunciations and vocabularies in Chongqing dialect scenarios, this paper constructs a multi-scenario and multi-speaker Chongqing dialect speech dataset. Based on the FunASR framework, we fine-tune the lightweight pretrained model SenseVoiceSmall. Four groups of controlled experiments are designed: baseline, SpecAugment augmentation only, domain hotword enhancement only, and their combination. Results show that the joint optimization strategy achieves the best performance: the model’s Average Correctness (Avg Corr) increases from 81.67% to 84.48%, and Average Character Error Rate (Avg CER) decreases from 25.31% to 20.09%. The recognition of colloquial expressions, unique vocabularies, and typical pronunciations of Chongqing dialect is significantly improved. Meanwhile, the model achieves a Real-Time Factor (RTF) of 0.005 with an average inference latency of 0.045 seconds per utterance, demonstrating high efficiency for lightweight deployment. This work provides a lightweight solution for Chongqing dialect speech recognition and a valuable reference for low-resource dialect research.
Pengcheng Zhou, Yanhua Liang, Dehong He et al.· International Conference on...· 0 citations
Speech enhancement models optimised for perceptual quality metrics such as PESQ do not necessarily improve automatic speech recognition (ASR) performance, as aggressive noise suppression can distort the phonemic cues that acoustic models rely on. We propose a time-domain speech frontend based on the Mamba selective state space model, trained exclusively with ASR-oriented loss functions to preserve phonetically discriminative structure rather than maximise perceptual quality. The model follows a U-Net encoderdecoder architecture with 4 × temporal downsampling and six stacked bidirectional Mamba blocks operating on the raw waveform, trained with a multi-component loss combining multiresolution spectral supervision, log-mel feature matching, speech correlation preservation, and ideal-ratio-mask spectral objectives. On VoiceBank-DEMAND, the proposed model achieves 10.26% WER with wav2vec2-base-960h and 7.68% with whisper-base, achieving statistically equivalent WER to SE-Mamba (10.30% / 7.73%) while producing a lower PESQ of 3.10 versus 3.69. Phoneme-class analysis confirms that the proposed model matches or outperforms SEMamba on five of seven articulatory classes, demonstrating that ASR-oriented training objectives achieve equivalent robustness to perceptual enhancement models without modifying the downstream ASR system.
Reshma K R, A. Vuppala· International Conference on...· 0 citations
India's linguistic landscape spans over 700 languages and thousands of dialects, yet the vast majority of automatic speech recognition (ASR) systems support only a small fraction of this diversity. We present SraVaani-1.0, a multilingual ASR model covering 65 Indian languages and dialects, many of which currently have no publicly available or competing ASR system. SraVaani-1.0 is built on a FastConformer architecture and trained from scratch through a three stage the first stage, we perform self-supervised pretraining on 31,255 hours of unlabelled speech from the VAANI corpus using a contrastive learning objective. In the second stage, we introduce an audio-image representation alignment stage that leverages the paired images and speech available in the VAANI corpus. This multimodal alignment encourages the speech encoder to learn semantically richer representations by exploiting the relationship between visual context and spoken content, thereby improving downstream recognition, particularly for low resource the final stage, the aligned encoder is fine-tuned end-to-end using a Hybrid Token-and-Duration Transducer (TDT)-CTC decoder on 31,263 hours of labelled multilingual Indian speech compiled from 24 public datasets spanning 65 languages and dialects. We evaluate SraVaani-1.0 against three state-of-the-art multilingual ASR systems across eight benchmarks. SraVaani-1.0 achieves the lowest word error rate (WER) on a large number of language-dataset pairs while remaining competitive with the best-performing systems on high resource importantly, it is the only open-source evaluated model that provides transcription capability for multiple low-resource and tribal Indian languages, which are assessed exclusively on the VAANI benchmark.
Sujith Pulikodan, A. Basu, J. Pavankumar et al.· 1 citation· ⚡1
Kazakh automatic speech recognition (ASR) presents a persistent challenge for large-scale multilingual models. This paper presents a systematic evaluation of 27 ASR pipeline configurations (three ASR models × three VAD methods × three post-processing strategies) on the Kazakh Speech Dataset (KSD), examining the contribution of model fine-tuning, voice activity detection (VAD) preprocessing, and large language model (LLM) post-correction and benchmarking the resulting pipelines against two non-Whisper foundation models. Language-specific fine-tuning reduces Word Error Rate (WER) from 43.20% (generic Whisper-large-v3) to 11.88% (Kazakh fine-tuned Whisper-turbo), a 31.32-percentage-point absolute reduction (72.5% relative; p < 0.001, bootstrap test); the effect persists after controlling for model size (generic Whisper-large-v3-turbo, 18.92%, vs. the same architecture after fine-tuning, 11.88%; p < 0.001). VAD preprocessing consistently degrades performance. Zero-shot post-correction with general-purpose LLMs yields no benefit and adds substantial latency: Gemma-2-9B and Qwen2.5-7B raise WER by 5.5 and 7.2 percentage points at real-time factors of 0.52 and 0.30, and a larger 32B model still degrades accuracy (+10.8 points), indicating that scale is not the limiting factor. Among all systems evaluated, a larger multilingual foundation model, SeamlessM4T-v2 (9.72% WER), outperforms the fine-tuned Whisper, showing that for Kazakh model coverage matters more than pipeline engineering. Character-level error analysis identifies systematic confusion between Kazakh-specific and Russian Cyrillic characters as a dominant error source. These findings establish that, for Kazakh under the evaluated conditions, model choice dominates pipeline add-ons: fine-tuning is essential, VAD and zero-shot LLM correction consistently hurt, and a strong multilingual model sets the best result; we further discuss the extent to which these conclusions extend to typologically similar Kipchak-Turkic languages.
Nursultan Nyssanov, L. Rzayeva, Alisher Batkuldin et al.· Information· 0 citations