Skip to content
Preprint

Multimodal Conversational Context for LLM-Based ASR: Data Construction, Training, and Benchmark

Sep 2026 · 0 citations · 28 references
Engineering

TL;DR

This work presents a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation and introduces MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios.

Abstract

Conversational context provides semantic and acoustic cues across turns for automatic speech recognition (ASR), but relying on historical transcripts can propagate recognition errors and discard pronunciation and speaker information. We present a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation. We construct dialogues around entities and their confusable forms and interleave historical user speech with assistant text responses for supervised fine-tuning. We also introduce MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios. Experiments with Qwen3-Omni and Step-Audio-2-mini reveal limitations in handling irrelevant and erroneous history and show that our data construction and training improve context utilization, with multimodal context achieving the highest overall entity recall on both models. Further experiments on accent, dialect, and target-speaker ASR demonstrate the value of historical speech. The benchmark data and evaluation code are publicly available at https://github.com/llh666521/MM-ContextASR.

View source

Similar papers

#natural language process... Open access Sep 2026

The Bairong System for MLC-SLM 2026: Dynamic Question-Aware Evidence Routing for Multilingual Conversational Speech Understanding

Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker-sensitive cues. We present the Bairong system for the MLC-SLM 2026 Challenge, where a diarization-ASR front-end produces speaker-attributed transcripts and a dynamic ev...

Shang-Kun Huang, Jun-Chao Hu, Hua Shen et al. · 1 citation
Open access Sep 2026

Context Projector: Complementary Keyword and Dialogue Context Embeddings for LLM-based ASR

Contact-center dialogue systems require both accurate transcripts and reliable recognition of business-critical entities. We introduce a hybrid keyword and context projector (CP) approach for LLM-based ASR. Rather than appending raw dialogue history, CP encodes previous turns into compact contextual tokens, complemente...

Esaú Villatoro-Tello, Sergio Gastón Burdisso, Shashi Kumar et al. · 5 citations
Book Open access Oct 2026

Backchannel-Aware Transcription for Multiparty Conversation: ASR Omissions, Acoustic Recovery, and Cross-Corpus Transfer

Collective-state research increasingly treats automatic transcripts as a substitute for raw audio. We show this is not free under one widely used pipeline: WhisperX omits approximately 48% of hand-labeled backchannels (“mm-hmm”, “yeah”) in close-talk multi-party recordings, and the omission replicates across held-out r...

Shan Ahmed Shaffi, M. J. Seikavandi, Anna Obara et al. · 1 citation

COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning

Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated speech typically relies on clear user-specified instructions. In natural conversations, speaking style should be naturally inferred from the preceding conversational con...

Wei-Zhen Bian, Si-Tong Cheng, Rong-Xiu Zhong et al. · 0 citations
Preprint Aug 2026

Phoneme-guided TTS augmentation for ASR: A unified pipeline and multilingual evaluation

A phoneme-guided text-to-speech augmentation pipeline for ASR that connects multilingual speech generation with candidate-text selection and reference-speech quality control, and proposes phoneme-frequency-guided selection (PFGS), which uses phoneme frequencies from real ASR training transcripts to prioritize candidate...

Zhen Wang, Tian-Rui Wu, Rong-Qi Han et al. · 0 citations
#natural language process... Preprint Oct 2026

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model, achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines.

Shi-Yue Zhu, Lian-Bo Liu, Sizhen Lyu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.