Skip to content

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

A dense, decision-level reward is introduced in which an LLM judge evaluates the necessity of each tool call, which effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.

Abstract

Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.

View source

Similar papers

Preprint Aug 2026

Joint Optimization of Tool Creation and Use for Large Language Model Agents

A reinforcement learning framework that jointly trains tool creation and tool use inside a single policy, with three separate reward axes that catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient.

Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen et al. · 2 citations
#artificial intelligence Preprint Sep 2026

Agents'Overreliance on Unreliable Tools

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with thre...

Hoyeol Yang, Woojung Song, Taewon Kim et al. · 2 citations
#artificial intelligence Review Sep 2026

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise.

Dong-Dong Zhang, Teng-Chao Lv, Yi-Ling Jia et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Constraint Tree Exploration for Learning from Language Feedback

This work introduces TRACE, an algorithm that organizes candidate constraints in a tree and tests each proposed refinement by generating actions that satisfy it, and evaluates TRACE across six language-feedback tasks, showing greater robustness than direct accumulation when the falsification detector remains reliable.

Shao-Ang Li, Daniel R. Jiang, Jian Li · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.