Infinite ResNets are Gaussian Mean ODEs
This paper obtains a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the $1/\text{depth}$ branch scaling that allows maximal (local) feature updates, and discusses Mixture of Expert models, where likewise the limit admits an intuitive Mean ODE description.