Calculator tool calling for improving mathematical reasoning on the Countdown task is studied, suggesting that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.
Abstract
Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.
Recent methods using Reinforcement Learning (RL) have improved Tool-Integrated Reasoning (TIR) by training large language models to learn end-to-end policies for multi-step tool usage, enabling them to solve complex tasks more effectively. Despite these advances, existing methods often suffer from overthinking at both the action and reasoning levels: models tend to invoke tools redundantly and generate excessively long reasoning trajectories, resulting in high computational cost. To address this, in this paper, we propose LightTIR, a dual-penalty reward framework, to achieve efficient TIR. For action efficiency, LightTIR estimates the marginal utility of each tool call through prefix-aligned counterfactual trajectories, encouraging calls that contribute meaningful information while penalizing low-utility or redundant invocations. For reasoning efficiency, LightTIR introduces a length-aware regularization term, adaptively penalizing intermediate reasoning steps that exceed the minimal effective trajectory required for correct prediction. Extensive experiments demonstrate that LightTIR can reduce redundancy and trajectory expansion while maintaining answer correctness, achieving more efficient RL-based TIR. Code is available at https://github.com/ekventitas/LightTIR.
Yichen Xiao, Siyu Gong, Linan Yue· Proceedings of the 32nd ACM...· 0 citations
Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. Tools written by our 4B models also lifted the performance of LFM-2.5-350M and Qwen3-30B-A3B under same reasoning tasks.
Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen et al.· 0 citations
SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments that combines a Gymnasium-compatible environment wrapper, a VERL-style rollout dataflow, group-relative policy optimization without a separate value model, and a sink-aware FlexAttention path designed to preserve model-specific sink scaling under causal and sliding-window masks is presented.
Zelei Cheng, Amritansh Mishra, Sambit Sahu et al.· 0 citations
Chemistry questions often demand exact computation and database lookups that a language model cannot supply from its parameters, so it must reach for external tools. Tool use here is a three-part problem: select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last. CheMatAgent, a previously published system, addresses this with hierarchical evolutionary MCTS: separate policy and execution models searching tool-call trees under two learned critics, one regressed partly onto GPT-assigned scores. We show that a single policy suffices. Our model interleaves reasoning, tool calls, and returns in one left-to-right generation, trained by a supervised warm-up and then outcome-level reinforcement learning against a programmatic reward read directly off the gold call chain, which leaves no learned critic and no judge in the training loop. On ChemToolBench multiple-tool comprehensive chemistry, on both backbones CheMatAgent use, we improve Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and by 3.7% and 3.9% on Llama-3.1-8B, compared with their strongest search configuration, at one model invocation per question, against a search whose cost grows with the tree; we also lead answer Pass Rate on Qwen-2.5-7B.
Armin Dariani, Si-Fan Wu, Bang Liu et al.· 0 citations
This work uses classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM to improve RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains.
Yuanning Feng, Niels Mündler-Sasahara, Mark Vero et al.· 0 citations
This work introduces StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment and substantially reduces the computational overhead of multimodal reinforcement learning.
