Author

Margaret Li

h-index 4 View on Semantic Scholar

We have 2 of 7 papers

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

#artificial intelligence Preprint Sep 2026

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits...

Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz et al. · 0 citations

#natural language process... Preprint Sep 2026

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Switch Distillation is proposed, a simple mid-training objective that distills on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal, and otherwise falls back to cross-entropy, which consistently outperforms existing distillation objectives across teacher sizes.

Jacqueline He, Howard Yen, S. Li et al. · 0 citations