Distilling Sequential Computation in Transformer Language Models
A method for distilling sequential computation by replacing spans of input tokens with collapsed representations, computed on the fly by a lightweight merge module, allowing pretrained models to operate on compressed inputs without architectural changes or re-training.