Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approxi...
Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava et al.· 0 citations
Post-training expert pruning is a promising deployment technique for Mixture-of-Experts (MoE) models. The calibration set is an integral part of the pruning process, however, its correlation with evaluation performance is largely misunderstood and unintuitive. In this work, we present an empirical study of the robustne...
Luis Frentzen Salim, Chia-Chun Wu, Nhiem Tran et al.· 0 citations
It is shown how fact-checking, a generally desirable behavior, can interfere with belief tracking in LLMs and how suppressing this attention at decoding time recovers accuracy only partially and only in some models, calling for future work on intervention methods.
Quang Minh Nguyen, Luis Frentzen Salim· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.