HyperTransfer is proposed, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself, to extend the framework to non-scale-invariant networks.
Abstract
Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.
This work introduces RODE, which gives the radial and directional components separate update rules and step sizes in the matrix Frobenius norm, and suggests that decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization.
Guo-Xiang Xu, Bince Qu, Qi Sun et al.· 0 citations
Results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics, and Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce.
Zi-Han Liu, Rui-Heng Zheng, Shaobo Zhang et al.· 4 citations
This work introduces Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models, and shows that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately unif...
Feng-Zhuo Zhang, Shu-Che Wang, Sheng-Gui Li et al.· 0 citations
Block normalization generally improves Adam and closes part of its gap to Muon across molecular force prediction on the rMD17 and MD22 datasets, QM9 molecular property prediction, and charged particle dynamics.
This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems.
Results support a split EoS picture for Muon: a stochastic loss-neutral edge survives, but it is not accompanied by a universal temporal-direction signature, and directions remain far from coherent reversal while training continues to improve.
Yan-Zhe Chen, Qifang Zhao, Xiao-Xiao Xu et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026