A dense, decision-level reward is introduced in which an LLM judge evaluates the necessity of each tool call, which effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.
Abstract
Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.
NLPG is proposed, an external policy-memory method for improving a fixed agent without changing its model parameters or program structure that outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average.
Xu Liu, Wen-Zhang Wei, Jun Cao et al.· arXiv.org· 0 citations
A reinforcement learning framework that jointly trains tool creation and tool use inside a single policy, with three separate reward axes that catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient.
Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen et al.· 2 citations
Results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone, and can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.
LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with thre...
Hoyeol Yang, Woojung Song, Taewon Kim et al.· 2 citations
CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise.
This work introduces TRACE, an algorithm that organizes candidate constraints in a tree and tests each proposed refinement by generating actions that satisfy it, and evaluates TRACE across six language-feedback tasks, showing greater robustness than direct accumulation when the falsification detector remains reliable.
Shao-Ang Li, Daniel R. Jiang, Jian Li· 0 citations