Parameter-efficient ASR for Tamil-English code-mixed speech using LoRA and QLoRA techniques

Abstract

For automated speech recognition (ASR) systems, code-switched speech in which speakers alternate between two or more languages in a single utterance presents substantial difficulties, especially when it comes to Tamil-English languages. By creating a robust code-switched corpus and a parameter-efficient ASR system specifically suited for Tamil-English multilingual settings, this work seeks to address the difficulties. We employ the Whisper ASR model, which has been optimized by parameter-efficient adaptation methods like Quantized LoRA (QLoRA) and Low-Rank Adaptation (LoRA). These techniques allow huge pre-trained models to be scaled and resource-efficiently customized to the distinct linguistic and phonetic features of Tamil-English code-switched communication. Additionally, we suggest optimization techniques to raise the recognition accuracy of Tamil-English code-mixed combi-nations. With a Word Error Rate (WER) of 32.4% and a Character Error Rate (CER) of 22.1%, a zero-shot Whisper model serves as the baseline. On comparing with Whisper(small, medium, Large) models the Whisper-Medium model, using LoRA (adapter rank = 16; target modules: q_proj, k_proj, v_proj, and out_proj) evaluated on the speaker-disjoint test set and this proposed approach achieved a WER of 25.9% and CER of 18.4%. Under the same configuration, QLoRA achieved a WER of 26.2% and CER of 18.6%, while reducing peak training VRAM compared with full fine-tuning, while lowering peak GPU memory usage from 14.9 to 9.6 GB.