Distilling Sequential Computation in Transformer Language Models
A method for distilling sequential computation by replacing spans of input tokens with collapsed representations, computed on the fly by a lightweight merge module, allowing pretrained models to operate on compressed inputs without architectural changes or re-training.
Zi-Xuan Lan, Jessica H. Yang, Yan-Hong Li et al.
· 0 citations