UltraData-Code: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data
Code generation has become a core capability of large language models (LLMs), and code data is central to the pre-training process that develops it. As obtaining public code of sufficient quality becomes more difficult, further progress cannot rely on corpus expansion alone. Tiered data management instead applies incre...