As large language models (LLMs) grow more powerful, their scale makes them costly to serve. Depth pruning offers a practical route to efficiency by directly reducing memory and latency without hardware-specific support. However, we observe that transformer layers exhibit markedly different task-dependent layer contribu...
Guan-Chen Li, Yi-Xing Xu, Xuan-Wu Yin et al.· 0 citations
AlTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format is presented.
Han-Jing Wang, Zhi-Tao Wang, Natalia Frumkin et al.· 0 citations
Zebra3D, a layerwise sparse-linear hybrid attention framework that improves the efficiency of pretrained 3D foundation models without training from scratch, achieves strong reconstruction and camera-pose accuracy at high attention sparsity, with training-free variants remaining close to their dense counterparts.
Utkarsh Saxena, Rui-Jing Yang, Mehdi Rezagholizadeh et al.· 0 citations
CompassPlay is introduced, a self-play method that rewards the proposer through gradient alignment that draws on a first-order approximation to learning progress and scores each eligible task without additional solver training.
Sophia Xiao Pu, Xi-Meng Sun, Jiang Liu et al.· arXiv.org· 0 citations
Instella-MoE is introduced, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs, establishing a strong, fully open foundation for efficient, high-performing MoE models and r...
Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.