Canary tools are introduced: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness, evidence that the probes measure reasoning, not phrase-spotting.
Abstract
Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness. A six-type taxonomy (semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps) turns a single"wrong tool"outcome into a multi-dimensional profile of how a model reasons about tools. We evaluate eight models -- six hosted and two 8B open-weight -- spanning three capability tiers, on 120 tasks across three canary-density conditions and three seeds (8,640 runs), plus a 2,880-run subtlety ablation. Task success is graded by a provider-independent judge, corroborated by a second independent judge (Cohen's kappa = 0.75). We report three findings. First, susceptibility drops sharply as models get more capable: the per-task canary susceptibility rate (CSR) ranges about 36x across models, lowest for Claude Opus 4.8 and highest for Llama 3.1 8B. Second, capability tier alone does not predict safety: the most susceptible hosted model is mid-tier, and within a provider the cheaper model can be the safer one. Third, the taxonomy is capability-stratified: capability mirages most reliably trap frontier models, while the other types are largely inert on strong models but fire on small open models, so they discriminate by capability rather than being weak. Softening each canary's give-away phrase leaves frontier CSR essentially unchanged, evidence that the probes measure reasoning, not phrase-spotting. Susceptibility also predicts task failure (Spearman rho = -0.34), while the most robust models are not significantly degraded by canary pressure. We release the framework, canary schemas, tasks, and logs.
Reliable function calling (a.k.a. tool use) is a core capability of LLM agents. However, existing evaluations insufficiently probe how robustness varies with task complexity. We introduce MultiCAT-Bench, the first benchmark focused on detailed categorization of tasks for assessing tool utilization, along with an approach for its automated generation, which utilizes the GPT-5 family. MultiCAT-Bench spans ten principled categories of difficulty with 3 789 test cases. Using this dataset, we evaluate ten LLMs from 9 model families. The analysis revealed that Recall metrics (overall ~72.4%, tool name identification ~81%, arguments ~90.4%) are lower than Precision (overall ~88.3%, tool name identification ~99.8%, arguments ~93%) across the examined categories. This indicates that models are less likely to extract relevant information, but when they do, they achieve higher accuracy. Regarding the selected categories, the greatest impact on models’ accuracy was exerted by the number of calls in the response (average drop by a factor of 1.59), parameter optionality status (by a factor of 1.37), and the number of parameters in the function (by a factor of 1.22). The Grok 4.1 Fast and GPT-5 Mini models achieve the best average accuracy, 83.8% and 83.7%, respectively, across the benchmark.
A. Vyatkin, A. Poptsov, V. Oliseenko et al.· IEEE Access· 0 citations
Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyond text generation. However, clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call. We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents, where a tool-calling agent is an LLM system that selects a tool, supplies structured arguments, and interprets its returned feedback. ToolRobustBench aligns four perturbation families with the tool-use pipeline: tool-interface, user-intent, tool-output/observation, and runtime-environment perturbations. It attributes failures to tool selection, schema grounding, argument binding, tool-output/runtime-feedback handling, and E2E task success. Experiments on 15,456 single-family instances across 7 models, 16 sampled local tools, 4 perturbation families, and 14 subtypes show high but non-uniform clean performance and substantial robustness degradation, with tool-output/observation perturbation the dominant bottleneck. Mixed-family experiments reveal non-additive failure patterns that are not explained by isolated single-family results. Thus, ToolRobustBench provides a deterministic and cascade-aware benchmark for diagnosing robustness beyond clean tool-calling accuracy;
It is shown that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents'informative bands.
A probe corpus of 42 retracted, fraudulent, and pseudoscientific papers is paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing, indicating an urgent need for guardrail infrastructure for scientific deployment of language models.
ToolFailBench is introduced, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate, showing that faithful tool use is not saturated.
This work measures a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context, on five open-weight models over contamination-free multi-step tasks.
Afiya Noorain, Subhranshu Mohanty, Amritesh Banerjee et al.· 0 citations