Experimental results show that VALL-E outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity and could preserve the speaker’s emotion and acoustic environment from the prompt in synthesis.
Text-to-speech systems are improving fast, but measuring how natural they sound still requires expensive human listening tests. Existing automatic methods struggle to generalise well across different datasets. We present QMOS, a MOS prediction framework that extracts hierarchical speech quality features from a frozen Qwen2-Audio large audio language model and combines them with layer-weighted WavLM SSL representations through a learned cross-attention fusion. This design captures both high-level semantic naturalness and lowlevel acoustic distortions in a unified model. On two standard benchmarks, SOMOS and BVCC, QMOS achieves competitive system-level SRCC of 0.923 and 0.921 on SOMOS and BVCC, respectively, while using no system-ID conditioning or listener embeddings. Cross-domain evaluation yields a competitive SRCC of 0.702, suggesting the learned representations generalise across acoustic domains.
Sri Ravi Sastry Kolluru, Charan Devarakonda, S. Radhe et al.· International Conference on...· 0 citations
Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.
Huaxuan Wang, Huimin Wang, Ruiyu Zhang et al.· 0 citations
In view of the problems such as pronunciation deviations, wrong stress placement, and abnormal prosody that occur frequently in German speech synthesis in practical applications, this paper builds a large-scale data-driven intelligent error correction model based on its own constructed large-scale data. Firstly, it integrates offline corpora and online logs, collecting approximately 2.1 million pieces of text and 1,300 hours of speech data. The offline corpora and user logs account for 61.5% and 38.5% of the total duration respectively. Through forced alignment and ASR playback, phoneme/word-level fine-grained error annotations are achieved, and text–phoneme–acoustic multimodal features and PCA reduction representations are constructed. On this basis, a "error detection sub-network + error correction generation sub-network" is attached to the acoustic layer of the German TTS baseline system. A joint loss combining weighted binary cross-entropy and L1/L2 combined regression is adopted for multitask end-to-end training on large-scale samples, and lightweight deployment is achieved through channel pruning, distillation, and quantization. Experimental results show that compared with the baseline system, the proposed method improves the error detection F1 score to 78.4% on the test set, reduces the MCD from 4.12dB to 3.76dB, increases the MOS from 3.85 to 4.21, and increases the RTF only from 0.78 to 0.83. This indicates that the intelligent error correction model significantly improves the objective quality and subjective listening experience of German speech synthesis without sacrificing real-time performance.
Xingyan Lu· The 2026 International Confe...· 0 citations
This work trains an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models.
Harsha Vardhan Khurdula, Abhinav Singh, Yoeven D. Khemlani et al.· 0 citations
IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information, improves average accuracy and improves a model already explicitly fine-tuned to prioritize acoustic evidence.
Yu-Han Huang, Chih-Kai Yang, Ke-Han Lu et al.· 0 citations
Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.
Qingjian Lin, Yuxin Li, Haoyang Zhang et al.· 1 citation