Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences
A tractable teacher--student model where a stable latent linear RNN generates trajectories and a sketched linear recurrent student is trained by safeguarded full-batch WSD gradient descent on next-token prediction is studied.
Zi-Yan Chen, Zhong-Zhu Zhou, Pei-Lin Liu et al.
· 0 citations