Linear RNNs offer linear-time sequence processing and constant-memory decoding, but their fixed-size recurrent states must accommodate all past key--value associations. Existing forgetting mechanisms and Delta Rule updates reduce interference by selectively clearing or correcting the state, yet earlier associations can...
This work shows why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples, and connects the three interventions on the same facts.
Yi-Xiao Chen, Ke Cheng, Jiang-Tao Guan et al.· 0 citations
Hybrid language models can improve capability as well as efficiency, raising the question of how architectural complementarity becomes learned computation. We examine the established induction roles of Carrying predecessor information, Matching a source by content, and Copying its value. How are these position-sensitiv...
Ke Cheng, Xin Xu, Yi-Xiao Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.