This work uses classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM to improve RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains.
Abstract
Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.
This thesis proposes a unified two-axis framework that organizes SFT and RL methods along a data axis (off-policy to on-policy) and a loss function axis (positive-only to positive-plus-negative to GRPO) and enables controlled ablations of individual components.
G. Kim, Chair Chenyan Xiong, Aditi Raghunathan· 0 citations
This work systematize the RL-for-LLM paradigm and provides a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants, and develops a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs.
Maciej Besta, Leonard Schmidt, Lara Nonino et al.· 0 citations
Reinforcement learning with verifiable rewards (RLVR) has been established as a viable paradigm for the post-training of Large Language Models (LLMs), including downstream tasks, such as Neural Machine Translation (NMT). With the latest research indicating that RLVR could be the preferred training method for translating legal documents due to the induced reasoning capabilities, it raises the question whether it is really attributed to the reasoning or more generally to the training paradigm. We investigate the importance of including the model's reasoning trace in the generated responses during both training and inference by systematically omitting it from one of the phases. Our experiments show that including the reasoning, specifically during inference, has a positive effect on the overall translation quality. Furthermore, we recognise that the reasoning leads to an increase in output tokens, hence we study the cost-quality tradeoff between the increased computational demands and the improved translation quality.
Michael Jungo, Aixiu An· Proceedings of the 2026 ACM...· 0 citations
Large Language Models (LLMs) trained using Chain-of-Thought (CoT) supervision have achieved state-of-the-art performance on complex reasoning tasks. However, the generation of long reasoning chains introduces substantial computational overhead during inference, limiting their deployment in low-latency and resource-constrained environments. This paper proposes AdaptiReason, a novel framework that dynamically compresses intermediate reasoning steps based on task difficulty and model confidence without requiring retraining of the underlying base model. AdaptiReason employs a lightweight difficulty estimator to determine the appropriate reasoning depth for each input, followed by a learned token-pruning policy that eliminates redundant or low-information reasoning steps. Experimental evaluation on the MATH, GSM8K, and ARC-Challenge benchmarks demonstrates that AdaptiReason reduces the average number of generated tokens by 3.7× while preserving 98.2% of the baseline reasoning accuracy. Furthermore, the proposed framework is model-agnostic and can be seamlessly integrated with instruction-tuned LLMs without requiring access to model parameters, relying solely on output logits for adaptive reasoning compression. The results demonstrate that AdaptiReason significantly improves inference efficiency while maintaining high reasoning performance, making it suitable for real-time and resource-constrained LLM applications.
V. A, Mithaguru, Amrita Kundu et al.· 2026 4th International Confe...· 0 citations
Variance Recovery Policy Optimization (VRPO) is introduced, which retains and progressively expands groups to recover informative signals from prompts that are difficult yet solvable and retains and progressively expands these groups to recover informative signals from prompts that are difficult yet solvable.
Jingqi Tian, Haoji Zhang, Lin Chen et al.· 0 citations
Empirical findings suggest that CudaPerf significantly outperforms strong baselines, including Qwen-3-32B and CUDA Agent by achieving up to 5X and 3.32X improvements in speedup, and 17%&7% improvements in correctness, respectively.
Q. I. Mahmud, Nesreen K. Ahmed, Ali Jannesari· 0 citations