PPL-Factory is proposed, a simple and interpretable data selection framework that combines task-aware perplexity-based scores and data budget-aware selection criteria that outperforms other state-of-the-art data selection methods using only $1\%$ of the training set.
Abstract
Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce the computational cost while preserving downstream performance. Many existing data selection methods rely on indirect heuristics, such as data quality, diversity or reasoning trace length. However, the effectiveness of these fixed criteria is task-dependent and difficult to generalize across diverse downstream tasks. Perplexity-based data selection provides a simple and model-aware solution to estimate the sample difficulty, but existing approaches typically score the entire training sequence and ignore the difference in learning objectives of language modeling and reasoning tasks. In this paper, we propose PPL-Factory, a simple and interpretable data selection framework that combines task-aware perplexity-based scores and data budget-aware selection criteria. Experiments on GSM8K demonstrate that PPL-Factory outperforms other state-of-the-art data selection methods using only $1\%$ of the training set. With $10\%$ of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 on GSM8K and 4.8 on MATH. Overall, our results demonstrate that task-aware and budget-aware perplexity-based selection provides an effective and applicable approach for efficient fine-tuning.
The transition from architecture-centric scaling to data-centric refinement has established high-quality data as a critical determinant of Large Language Model performance, particularly for complex reasoning and instruction following. However, effective data selection remains a persistent bottleneck: simple heuristic filters often fail to capture multifaceted data features (e.g., reasoning depth and diversity), while advanced model-based scoring methods typically prioritize isolated quality dimensions, failing to provide a holistic assessment. To address these challenges, we propose R-Select, a robust and scalable framework that optimizes data selection with 30 distinct quality metrics. Recognizing that optimizing such a high-dimensional feature space is non-trivial, R-Select introduces a novel hierarchical optimization strategy. This approach structurally decomposes the search problem by first clustering correlated metrics into functional groups based on statistical dependencies. It then executes a two-stage optimization process: performing intra-group refinement to maximize local representational power, followed by inter-group integration to balance global quality domains. Crucially, to ensure computational efficiency, R-Select employs a low-resource proxy strategy, utilizing a lightweight model on a small data subset to learn an optimal selection policy that is transferable to the target model. Extensive experiments demonstrate that R-Select consistently outperforms both heuristic baselines and model-based methods, offering a robust solution for high-quality data curation.
Xin Gao, Xiaoyang Wang, Yun Zhu et al.· Proceedings of the 32nd ACM...· 0 citations
Results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance, and show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
Niqi Lyu, Pengtao Shi, Wei Qiu et al.· 0 citations
The Powered Length Penalty (PLP) is proposed, an adaptive regularizer that penalizes redundancy in short sequences while gradually reducing penalties for longer sequences, preserving deep reasoning.
Zehui Ling, Deshu Chen, Hongwei Zhang et al.· 0 citations
Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers. In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection. Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at https://anonymous.4open.science/r/MCFMOE.
Shuhan Huang, Naifan Zhang, Yuanbo Tang et al.· 0 citations
AssistEM, a framework for efficient LLM adaptation to EM via principled data selection, demonstrates that selective fine-tuning not only accelerates adaptation but also improves training efficiency (requiring fewer GPU hours), enabling open-source LLMs to rival–and in some cases outperform–closed-source models.
John Bosco Mugeni, S. Lynden, Toshiyuki Amagasa et al.· International Journal of Dat...· 0 citations
Although existing instruction data selection methods have introduced various metrics, the inherent complexity of real-world datasets makes it impractical for any single metric to generalize across all scenarios. Developers are thus often forced to manually inspect data and craft heuristic rules for each new application---a tedious and error-prone process. In this paper, we propose a paradigm shift from manual configuration to automated orchestration via the Instruction Data Selection Agent (DataMaster), which interprets user intent and autonomously composes optimal selection strategies. By allowing users to specify data needs through natural language descriptions, DataMaster simplifies data curation and removes the burden of manual strategy design. Extensive experiments across the math, medical, and code domains show that DataMaster outperforms static baselines in most settings and surpasses full-pool training in a substantial number of cases. The implementation of DataMaster and the scripts needed to reproduce the reported pipeline are publicly available at https://github.com/nju-websoft/DataMaster.
F. Zhou, Qiaosheng Chen, Zixian Huang et al.· 0 citations