Pipelining the Memory Hierarchy for Ultra-Long-Context Language Model Training
Long-context language models are attractive because they can read whole documents, code repositories, conversations, and scientific sequences. However, a long inference window is not the same as a model that was trained to use millions of tokens reliably. Direct long-context training is blocked by temporary memory spik...