This work proposes "Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays.
Abstract
The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's"Memorization Window"signals derived from loss retention dynamics and downstream evaluation scores, we propose"Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.
This work proposes TTCD, a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use, and focuses on an in-place variant: In-Place TTCD, which uses the existing MLP parameters as the fast weights.
Zixuan Wang, Xingyu Dang, Ruiming Zhu et al.· 0 citations
Reinforcement learning from human feedback (RLHF) has become the cornerstone of aligning large language models (LLMs) with human intent. Yet a fundamental question remains unaddressed: how should training data be scheduled when both the model's capabilities and the utility of data are constantly evolving? Current pipelines rely on fixed or uniform sampling, treating data value as static, an assumption we demonstrate to be fundamentally flawed. We identify and formalize the Distribution-Value Coevolution principle: the training value of data is not intrinsic, but emerges dynamically from the interaction between data characteristics and the model's evolving capability boundary. What is highly informative at one stage may become redundant, or even detrimental, at another. This insight demands a paradigm shift from static to adaptive curriculum design. We operationalize this principle through a unified framework with three components: (1) distribution-level organization that groups training data into coherent distributions; (2) sliding-window influence estimation that continuously tracks each distribution's evolving training value; and (3) bandit-guided scheduling that adaptively allocates resources with provable exploration-exploitation guarantees. Experiments show that this approach yields measurable improvements, with up to a 57.1% relative (or 8.9% absolute) improvement on AIME24 for Llama3.2-3B, and gains also observed for models ranging from 1B to 7B parameters.
Zairun Yang, Yanbo Yang, Chenyi Zhou et al.· Proceedings of the 32nd ACM...· 0 citations
Spaced Repetition Training (SRT) is introduced, a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm, and preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade.
A simple method, Self-Guided TTT (S-TTT), which improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
Xinyu Zhu, Zhenqin Xu, Xiaohan Wei et al.· 0 citations
Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the expressivity of per-token update dynamics with the hardware efficiency of chunk-wise approximations. We propose E$^2$-TTT (Expressive and Efficient TTT) to bridge this gap. Under the standard approximation of taking gradients at the chunk-start weights, we derive a closed-form state transition that exactly reproduces the chunk-end fast-weight and momentum states of the per-token recurrence. This enables fully parallelized chunk-level training while preserving the temporal structure of the update rule that prior chunk-wise methods discard. We validate E$^2$-TTT by training models up to 1.3B parameters from scratch. It performs on par with previous TTT and hybrid attention baselines in language modeling while outperforming them on in-context retrieval. Its advantage is most pronounced in length extrapolation: on the standard ``Needle in a Haystack''passkey test, it retains over 90% accuracy at $8\times$ the training context length. Meanwhile, E$^2$-TTT can match the training throughput of efficient chunk-wise methods, demonstrating that it effectively reconciles expressivity with efficiency. The code is available at https://github.com/zeyun-zhong/E2-TTT.
Zeyun Zhong, Joya Chen, Manuel Martín et al.· 1 citation
The results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.