LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residua...
Ze-Feng He, Yun-Kai Chai, Xiao-Yang Chen et al.
· 0 citations