Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits...
Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz et al.
· 0 citations