This work presents a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation and introduces MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios.
Abstract
Conversational context provides semantic and acoustic cues across turns for automatic speech recognition (ASR), but relying on historical transcripts can propagate recognition errors and discard pronunciation and speaker information. We present a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation. We construct dialogues around entities and their confusable forms and interleave historical user speech with assistant text responses for supervised fine-tuning. We also introduce MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios. Experiments with Qwen3-Omni and Step-Audio-2-mini reveal limitations in handling irrelevant and erroneous history and show that our data construction and training improve context utilization, with multimodal context achieving the highest overall entity recall on both models. Further experiments on accent, dialect, and target-speaker ASR demonstrate the value of historical speech. The benchmark data and evaluation code are publicly available at https://github.com/llh666521/MM-ContextASR.
Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker-sensitive cues. We present the Bairong system for the MLC-SLM 2026 Challenge, where a diarization-ASR front-end produces speaker-attributed transcripts and a dynamic ev...
Shang-Kun Huang, Jun-Chao Hu, Hua Shen et al.· 2nd Workshop on Multilingual...· 1 citation
Contact-center dialogue systems require both accurate transcripts and reliable recognition of business-critical entities. We introduce a hybrid keyword and context projector (CP) approach for LLM-based ASR. Rather than appending raw dialogue history, CP encodes previous turns into compact contextual tokens, complemente...
Collective-state research increasingly treats automatic transcripts as a substitute for raw audio. We show this is not free under one widely used pipeline: WhisperX omits approximately 48% of hand-labeled backchannels (“mm-hmm”, “yeah”) in close-talk multi-party recordings, and the omission replicates across held-out r...
Shan Ahmed Shaffi, M. J. Seikavandi, Anna Obara et al.· Companion Publication of the...· 1 citation
Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated speech typically relies on clear user-specified instructions. In natural conversations, speaking style should be naturally inferred from the preceding conversational con...
A phoneme-guided text-to-speech augmentation pipeline for ASR that connects multilingual speech generation with candidate-text selection and reference-speech quality control, and proposes phoneme-frequency-guided selection (PFGS), which uses phoneme frequencies from real ASR training transcripts to prioritize candidate...
Zhen Wang, Tian-Rui Wu, Rong-Qi Han et al.· 0 citations
Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model, achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines.
Shi-Yue Zhu, Lian-Bo Liu, Sizhen Lyu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.