No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we de...
Lewis Mitchell
· 0 citations