This paper introduces Spark-Audio 1.0, a speech foundation model trained on domestically produced computing infrastructure. The model accepts text and speech inputs and generates text outputs. It supports multilingual (99 languages) and multi-dialect (202 dialects) speech recognition, speech translation, audio percepti...
Chen-Yue Zhang, Cong Liu, Ding-Shu Tian et al.· 0 citations
Scaling Large Language Models (LLMs) via Mixture-of-Experts (MoE) enables massive parameter growth with nearly constant per-token computation. However, further scaling the parameter count requires increasingly sparse routing, where expert load imbalance becomes more severe. This imbalance reduces parameter utilization...
Peng Jin, Zi-Han Qiu, Ze-Kun Wang et al.· 0 citations
This work introduces Coarse-to-fine Error-aware Dynamic Attention Routing (CEDAR), a coarse-to-fine method that keeps the language model frozen while preserving global coverage and derives an output-error bound governed by within-chunk key/value dispersion and uses it to allocate a variable refinement budget.
The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests.
Zi-Han Qiu, Ze-Kun Wang, Xiao Li et al.· 42 citations· ⚡7
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.