Linger and Lose: Knowledge Collapse in Low-Bit Language Models
This work measures knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content and suggests judging low-precision training by retained capacity during training rather than final loss.