Skip to content
Preprint

Tool Retrievers Are Underestimated: Annotation Expansion Reveals True Capability

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

The proposed ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones, and confirms that the one-to-one problem extends beyond tool retrieval.

Abstract

In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Plan or Retrieve First? A Scale Study of Tool Discovery Architectures

Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate near...

Ming-Fei Zhang, Amit Surana, Ziauddin Majid et al. · 0 citations
#artificial intelligence Preprint Sep 2026

A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis

A novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data is introduced, suggesting that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.

Muhammad Jawad Chowdhury, Md. Sakib Khan · 1 citation
Book Open access Oct 2026

Can Large Language Models Identify Technical Terms in Documents Like Human Annotators?

Processing technical terms in documents is essential for digital libraries and knowledge organisation, supporting applications such as scholarly search, information retrieval, and domain terminology construction. While document-level term identification, a sub-task of automatic term extraction (ATE), has been tackled a...

Rei Miyata · 0 citations
Book Open access Oct 2026

LLM-Based Semantic Metadata Enrichment for Findable Scientific Data Collections

This work generates natural-language descriptions for opaque columns with a language model at ingestion, from their names, statistics, and sample values, and stores them as curator-reviewable catalog metadata for search and a natural-language visual-discovery interface.

Caio N. Maia, A. N. Maia, Pedro L. P. Corrêa · 0 citations
Preprint Aug 2026

Risk-Aware Reranking for Agentic Tool Retrieval

It is indicated that retrieval-stage filtering can reduce the candidate action space exposed to agents before execution, complementing downstream tool-use safeguards, and providing a conservative operating point for safety-critical deployments.

Qinfei Li, Xiaoxuan Dong, Jin Zhang et al. · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.