Skip to content

SoFT: Soft Targets for Generalizable LLM Fine-Tuning

Sep 2026 · arXiv.org · Vol abs/2609.32493 · 1 citation · 56 references
Computer Science

TL;DR

This work proposes soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.

Abstract

Distillation enables student language models to acquire new capabilities from expert teachers. However, integrating knowledge from multi-teacher, multi-domain demonstrations into a single student remains challenging. We study supervised fine-tuning (SFT) in this setting, where students must acquire diverse capabilities while maintaining generalization beyond the training tasks. Our experiments reveal varying trade-offs between in-distribution learning and out-of-distribution generalization across SFT methods, motivating more explicit control over this balance. To this end, we propose soft-target fine-tuning (SoFT) to balance learning from teacher demonstrations with retaining the Base model's existing capabilities. SoFT sets a minimum target probability for each demonstrated token while making the smallest KL change to the Base distribution. The resulting objective couples learning from demonstrations with adaptively weighted regularization toward the Base model. We further use domain-specific gradient budgets to control this balance and determine a probability threshold for each trajectory. Experiments on mixed-domain reasoning and agentic tasks show that SoFT achieves the best overall performance among the compared methods, with improvements in both in-distribution capability acquisition and out-of-distribution generalization.

View source

Similar papers

Oct 2026

Beyond frozen teachers: co-training and co-distillation for task-specific fine-tuning

Across the seven tasks and hyperparameter configurations the authors consider, CTCD is competitive with vanilla distillation and outperforms supervised fine-tuning alone on average; it transfers to task-specific fine-tuning as a viable distillation objective, but not as an improvement over the standard formulation.

Nikhil Trivedi, Karthik Ravi, Mohammad Ali Shehral et al. · 0 citations
Preprint Oct 2026

Self-Distillation for Supervised Fine-Tuning

Gold-Conditioned Distillation is proposed, a method that turns expert demonstrations into model-native supervision and improves new-task accuracy over SFT and generally reduces degradation on general benchmarks.

Wenxuan Zhu, Bang An, Shu-Ming Liu et al. · 0 citations
#machine learning Preprint Nov 2026

Self-Privileged Fine-Tuning

Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typ...

Yang Hu, Xue-Yu Zhou, Yue Wu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Rethinking Self-Distillation for Multi-Teacher Capability Merging

Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inferen...

Roy Xie, Dan Friedman, Feng Nan et al. · 1 citation
#machine learning Preprint Oct 2026

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, op...

Si-Qi Zhu, Suo-Zhi Huang, Kai-Xuan Zhang et al. · 3 citations
Preprint Oct 2026

SRA: Selective Residual Adaptation for LLM Reasoning Enhancement

Knowledge distillation transfers reasoning capabilities from large language models to lightweight students. However, distilled small models still struggle with multi-step reasoning. We observe that the distribution divergences between teacher and student models along reasoning paths are highly sparse, and target tokens...

Zhen-Xing Dou, Kun-Hua He, Ming-Yang Yi et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.