We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reason...
Xiang Li, Cheng-Ting Yu, Jun-Gang Li et al.· 0 citations
SMAT (Simple MAT) is introduced, which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise, and improves the mean score across five merging methods.
Yang-Gan Gu, Yuan-Yi Wang, Zhen Li et al.· 0 citations
A systematic analysis of expert routing patterns in MoE models reveals Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets, and proposes RISE, a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks.
Kening Zheng, Wei-Chieh Huang, Jiahao Huo et al.· arXiv.org· 4 citations· ⚡2
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.