Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
This work addresses the adverse interactions between sparsity and data repetition, and presents evidence for the core mechanisms of overfitting and its potential remediation, and suggests promising avenues for future methods to reduce over-specialization in model parameters by disrupting memorization patterns.
Atindra Jha, Margaret Li, J. Leskovec et al.
· 2 citations