On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak...
Ming-Yu Cao, Yao-Kai Jin, Shi-Wei Liu et al.· 0 citations
ActTail is proposed, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory that improves both perplexity and downstream task performance at high sparsity compared to uniform allocation.
As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training? We introduce DataSense-Bench to study this capability through the fundamental problem...
Yudi Zhang, Ming-Yu Cao, Lu Yin et al.· 1 citation
In this paper, we argue that Looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We first introduce an interesting phenomenon we observed in Looped Transformers, called latent overthinking: noisy state updates overwrite correct intermediate deduc...
Peng-Xiang Li, Dilxat Muhtar, Di He et al.· 1 citation
Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that th...
Di He, Peng-Xiang Li, Da Chang et al.· 3 citations
Unified multimodal models combine understanding, generation, and editing within a single network, offering a promising foundation for versatile multimodal applications. However, growing multimodal contexts make KV cache storage and access increasingly costly. Existing KV cache compression methods are typically tailored...
Wan-Qi Yang, Yue-Xiao Ma, Mei Xie et al.· 1 citation
FlashLoop is introduced, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization, substantially improving the practicality of scaling Looped Transformers to greater computational depths and longer context.
Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through t...