2026· Annual Meeting of the Association for Computational Linguistics· pp. 14866-14898· 0 citations· 39 references
Computer Science
TL;DR
The Invo-cation Refiner is introduced, a specialized post-processing module designed to enhance the tool-use reliability of base LLMs without directly training on them, and is highlighted as a plug-and-play solution for improving the operational reliability of LLM-based agents.
Abstract
Large Language Models (LLMs) have shown remarkable capabilities in Tool-Integrated Reasoning (TIR). However, the practical application is often hindered by frequent errors in tool invocations, such as incorrect parameters or malformed formats. Prevailing training paradigms, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), can mitigate these issues but require modification on the base LLM. This lack of modularity necessitates extensive retraining when deploying the system across different base models. To address the limitation, we introduce the Invo-cation Refiner, a specialized post-processing module designed to enhance the tool-use reliability of base LLMs without directly training on them. The Refiner takes the output from a frozen upstream LLM and the user’s query as input, performing independent reasoning to rectify the invocation. We construct a dedicated training dataset and train this module using an advanced RL algorithm. On a diverse set of tool-use and reasoning benchmarks, our Re-finer improves task completion rates and invocation accuracy over the raw outputs of various upstream LLMs. This highlights our Refiner as a plug-and-play solution for improving the operational reliability of LLM-based agents. We release our code to facilitate future research.
The results show that metainfer organizes generation constraints, validation feedback, and knowledge consolidation into a continuous closed loop, enabling runnable customized inference solutions to be generated from explicit knowledge.
Zhenwen Miao, Honglin Wang, Mingheng Mi et al.· 0 citations
Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyond text generation. However, clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call. We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents, where a tool-calling agent is an LLM system that selects a tool, supplies structured arguments, and interprets its returned feedback. ToolRobustBench aligns four perturbation families with the tool-use pipeline: tool-interface, user-intent, tool-output/observation, and runtime-environment perturbations. It attributes failures to tool selection, schema grounding, argument binding, tool-output/runtime-feedback handling, and E2E task success. Experiments on 15,456 single-family instances across 7 models, 16 sampled local tools, 4 perturbation families, and 14 subtypes show high but non-uniform clean performance and substantial robustness degradation, with tool-output/observation perturbation the dominant bottleneck. Mixed-family experiments reveal non-additive failure patterns that are not explained by isolated single-family results. Thus, ToolRobustBench provides a deterministic and cascade-aware benchmark for diagnosing robustness beyond clean tool-calling accuracy;
This work introduces ToolAtlas, a graph-based framework that builds a persistent provider-side tool memory of tool capabilities, failure boundaries, and cross-tool compositions through execution-verified probing and establishes provider-side tool memory as an effective and reusable paradigm for tool servers.
Yue Fang, Zhibang Yang, Fangkai Yang et al.· 0 citations
This work operationalizes constructive specification with constructive specification, which builds hierarchical capability representations from limited profiling over diverse benchmarks, using an optimism-guided profiler that prioritizes informative regions and prunes low-utility areas with guarantees, and enables plug-and-play identification of lightweight agents.
Jian-Dong Liu, Zi-Chen Zhao, Haodong Sun et al.· Proceedings of the 32nd ACM...· 1 citation
Large Language Models (LLMs) trained using Chain-of-Thought (CoT) supervision have achieved state-of-the-art performance on complex reasoning tasks. However, the generation of long reasoning chains introduces substantial computational overhead during inference, limiting their deployment in low-latency and resource-constrained environments. This paper proposes AdaptiReason, a novel framework that dynamically compresses intermediate reasoning steps based on task difficulty and model confidence without requiring retraining of the underlying base model. AdaptiReason employs a lightweight difficulty estimator to determine the appropriate reasoning depth for each input, followed by a learned token-pruning policy that eliminates redundant or low-information reasoning steps. Experimental evaluation on the MATH, GSM8K, and ARC-Challenge benchmarks demonstrates that AdaptiReason reduces the average number of generated tokens by 3.7× while preserving 98.2% of the baseline reasoning accuracy. Furthermore, the proposed framework is model-agnostic and can be seamlessly integrated with instruction-tuned LLMs without requiring access to model parameters, relying solely on output logits for adaptive reasoning compression. The results demonstrate that AdaptiReason significantly improves inference efficiency while maintaining high reasoning performance, making it suitable for real-time and resource-constrained LLM applications.
V. A, Mithaguru, Amrita Kundu et al.· 2026 4th International Confe...· 0 citations
Many of the domain-specific languages we use every day are not written to files but typed at an interactive prompt, e.g., database shells, cloud CLIs, and in-house analytics consoles. For these REPL-driven command languages, autocomplete is, we argue, not a polish feature but a core usability requirement --and yet they are precisely the languages whose authors often have the fewest resources to invest in tooling. Designing a new programming language tailored to specific domain challenges can be both powerful and rewarding. However, a major hurdle for adoption among users is, in our experience, the lack of tooling support, particularly features like autocomplete that enhance usability and reduce the learning curve. In this paper, we present a fresh perspective that turns parser errors -- traditionally seen as roadblocks -- into opportunities for generating valid, context-aware autocomplete suggestions. By leveraging the parser's built-in feedback mechanisms, our method offers a lightweight, adaptable, and simple solution, enabling domain-specific languages to provide basic development assistance with minimal overhead. The technique fits the REPL-driven DSL genre by design: inputs are modest-size single statements, the cursor sits at the end of the line, and parser-error recovery is unnecessary. We apply the technique to DPella DSL, a production REPL-driven DSL whose grammar comprises 43 top-level command constructors, 130 reserved keywords, and 22 labelled syntactic categories that drive identifier completion.
M. Gissurarson, Elisabet Lobo-Vesga, Alejandro Russo· Proceedings of the 19th ACM...· 0 citations