Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
Large Language Models (LLMs) are now routinely trained using synthetic data, since high-quality human data has been exhausted by the ever increasing needs of larger and larger models. However, recursive training on synthetic data frequently induces model collapse, a degenerative feedback loop where models progressively...
Matteo Marchi, João Pedro Silvestre, Bahman Gharesifard et al.
· 0 citations