Aug 2026· Electronics· Vol 15, pp. 3605· 0 citations· 6 references
TL;DR
An AS-Split Conformer–Mamba framework that decouples local and global modeling into two explicit stages and achieves consistent improvements over strong baselines is proposed.
Abstract
Automatic speech recognition (ASR) systems based on Conformer architectures achieve strong performance by jointly modeling local acoustic patterns and global contextual dependencies. However, their interleaved convolution–attention design leads to progressive entanglement of fine-grained acoustic features and global semantic representations, which weakens monotonic alignment in speech recognition and degrades performance in long utterances. To address this limitation, we propose an AS-Split Conformer–Mamba framework that decouples local and global modeling into two explicit stages. First, a stage-wise encoder is introduced, where a dedicated local modeling stage extracts phonetic-level acoustic features using SE-enhanced convolution, followed by a global modeling stage that captures long-range dependencies via multi-head self-attention and temporal convolution. Second, a Transition Fusion Block (TFB) is designed as an adaptive transition module that transforms local acoustic representations before they enter the global modeling stage. Third, intermediate CTC supervision is introduced to explicitly strengthen monotonic alignment at shallow representations. Finally, a hybrid Transformer–Mamba decoder is adopted, in which the Mamba block provides O(N) state-space computation within the replaced FFN sublayer while retaining Transformer attention mechanisms for acoustic–text alignment. Experiments conducted on AISHELL-1, THCHS-30, and ST-CMDS demonstrate that the proposed method achieves consistent improvements over strong baselines. On AISHELL-1, our model reduces Character Error Rate (CER) from 5.7% to 4.8% and Sentence Error Rate (SER) from 24.8% to 20.5%, while maintaining competitive computational efficiency.
Conventional Speech Named Entity Recognition (SNER) typically relies on cascaded ASR (Automatic Speech Recognition)+NER (Named Entity Recognition) pipelines, which are hindered by error propagation and the underutilisation of acoustic cues. We propose an end-to-end Chinese SNER framework using Residual Vector Quantisation (RVQ) and deep acoustic--semantic fusion. The model extracts speech representations via a frozen Wav2Vec2-XLSR encoder, employing an RVQ-based bottleneck to reconstruct continuous quantized features that regularize the acoustic space and preserve semantic content.
A Transformer decoder, trained with a joint CTC-attention objective, performs transcription while a gated deep-fusion mechanism integrates an external GPT model for linguistic consistency. For NER, a bidirectional multimodal fusion module aligns acoustic and semantic features before a GlobalPointer head performs span-level prediction. Experiments on AISHELL-NER and CNERTA yield F1-scores of 90.91\% and 81.44\%, respectively, outperforming pipeline, multimodal, and E2E baselines. These results demonstrate that deep bidirectional interaction between quantized acoustic streams and semantic contexts is essential for mitigating ASR error propagation and achieving robust Chinese SNER.
Yaoqiang Zhou· Poster Volume 0007 The 2026...· 0 citations
Attention-based encoder decoder models remain a popular choice for state-of-the-art automatic speech recognition (ASR). These models combine a powerful audio encoder that extracts rich acoustic features with a decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe a systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the later layers largely focus on merging acoustic and textual information for the final predictions. Leveraging this key insight, we propose B LOCK D ECODER , a novel decoder architecture comprising two distinct components: a text encoder that is purely text-based, and a M ERGER that combines information from the audio encoder and text encoder to generate output tokens. Unlike traditional decoders, the M ERGER autoregressively predicts a sequence of K tokens within a block of size K , while relying on the same precomputed contextual information from both text and audio encoders across the block. This design choice allows for the efficient reuse of encoder representations. The separation of the decoder into the text encoder and the M ERGER promotes modularity and more flexible control of parameters via the number of text encoder and M ERGER layers. As a result, B LOCK D ECODER yields a significant speedup ( ∼ 2 x) compared to traditional decoders, across diverse datasets, languages, and speech tasks, without any degradation in performance. The code is available at https://github.com/csalt-research/blockdecoder .
Darshan Prabhu, P. Jyothi· Neural Information Processin...· 0 citations
DELTA-TTS is introduced, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding and produces sharper text--speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation.
Junwon Moon, Yejin Lee, Seungbeom Kim et al.· 0 citations
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Qingjian Lin, Yuxin Li, Haoyang Zhang et al.· 1 citation
This work trains an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models.
Harsha Vardhan Khurdula, Abhinav Singh, Yoeven D. Khemlani et al.· 0 citations
Low-resource languages continue to face significant challenges in automatic speech recognition (ASR), especially those with limited annotated corpora and significant variant variation. The Amazigh language family, which is spoken throughout North Africa, has very little digital infrastructure and is severely underfunded. In this paper, we present a multi- variant Amazigh ASR system that integrates linguistically-informed post-processing correction and audio enhancement preprocessing with refined multilingual self-supervised models is presented. We address three key issues: (1) acoustic degradation in field recordings; (2) substantial inter-variant variation; and (3) extreme data scarcity. Our enhanced pipeline adds a semi-supervised learning framework that includes: (1) data augmentation through synthetic speech generation and aggressive spectral perturbation, which increases the training corpus to 15 hours; (2) self-training on unlabeled data using pseudo-labeling; (3) VoiceFixer-based audio restoration; and (4) hybrid n-gram/Levenshtein post-correction. WER of 18.7% ± 2.3% (95% CI), CER of 9.4% ± 1.6%, and PER of 12.1% ± 1.9% show statistically significant improvements, with a relative WER reduction of 34.2% compared to baseline Whisper (p < 0.001). Component contributions are quantified by ablation studies: audio preprocessing results in a relative improvement of 8.3%, while post-processing adds 12.7% reduction. In addition to offering a broadly applicable framework for the preservation of low-resource languages, this work sets new state-of-the-art for Amazigh ASR.
Youness Chaabi, Mohamed Oubenal· 2026 6th International Confe...· 0 citations