OO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, and outperforms every evaluated released learned drafter in each comparable cell, while the same sidecar improves on ToolSpec by 34.1% on average.
Abstract
LLMs generate tool calls token by token, even though the function choice and argument values can often be predicted in parallel from the request and tool schema. ToolSpec reduces this cost by drafting schema tokens and retrieving earlier calls, but cannot propose request-specific values absent from either source. We present OoO-Spec, which computes these missing semantics out of order. At request arrival, a Qwen3-0.6B sidecar predicts the function choice and all schema-defined argument slots in one parallel request-level wave while the target begins ToolSpec decoding. The runtime joins the slot values, renders the resulting call as text, and exposes it to subsequent candidate-construction rounds. The target polls without blocking, re-tokenizes a ready hint with its own tokenizer, and remains the sole verifier and commit authority. The sidecar is trained once with LoRA on Qwen2.5-32B teacher traces and used unchanged across Qwen2.5, Qwen3, and Llama targets, without target-specific drafter training. Across seven fully ranked targets and three benchmarks under greedy batch-one decoding, OoO-Spec is fastest among all evaluated methods in all 21 target-benchmark cells, reaching 2.46x-5.34x over autoregressive decoding with an unweighted mean of 3.89x, versus 2.95x for ToolSpec. It also outperforms every evaluated released learned drafter in each comparable cell. Across Qwen3-4B, 8B, 14B, and 32B targets, the same sidecar improves on ToolSpec by 34.1% on average. Its compact semantic payload averages 85 bytes per request excluding protocol metadata, supporting effective split-GPU overlap.
The Model Context Protocol (MCP) standardizes how servers expose data and tools to Large Language Models (LLMs). A common server design embeds frequently used reference data, such as identifier lookup tables, directly in the server instructions: the system-prompt text a server hands to the host application. When a query concerns an entry of the embedded table, the model can act on it immediately instead of re-discovering the same information through a search tool. We test whether client LLMs actually consume such instruction-embedded data, reporting a 54,000-trial study across 24 LLMs (9 Claude, 6 Gemini, 9 GPT) on a production legal-information MCP server. A diagnostic condition that removes the competing search tool shows that failures are dominated by behavioral preference rather than missing capability. With search unavailable, 23 of 24 models read the embedded data reliably (hit ratio at least 98%); with a search tool merely present, 9 models drop below 15%. A 2^3 factorial analysis of three instruction-level interventions reveals strong interaction effects: combining all three restores at least 86% for 20 of 24 models, but individual interventions can backfire for specific model families. Per-server prompt engineering is therefore a workaround rather than a fix; we argue that MCP host applications should provide an explicit mechanism that places server instructions ahead of tool selection in the client LLM's deliberation.
Minhan Cho, Soyoung Park, Kihyeon Jeong et al.· 0 citations
This work empirically compare programmatic tool calling to native JSON tool calling across 14 language models on BFCL v4 and demonstrates that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.
Ishan Patel, Sahil Sen, Elias Lumer et al.· 0 citations
Function calling enables Large Language Models (LLMs) to orchestrate backend services by invoking structured tools, extending their role from text generators to intelligent agents. While prior work evaluates tool-use accuracy, there is limited empirical evidence on the backend-level performance trade-offs between LLM-driven orchestration and traditional deterministic implementations in enterprise systems. This study presents a controlled, reproducible benchmark comparing Javabased deterministic backend logic with LLM function-calling pipelines using locally deployed open-weight models (Mistral 7B, Llama 3.1 8B, and Qwen 2.5 7B). Across three representative enterprise scenarios, data retrieval, data normalization, and command execution, we measure accuracy, latency, error rate, token efficiency, and development effort. Results show that deterministic implementations achieve near-instantaneous execution (0.2-1.8 ms) with 100% reliability, while LLM-driven approaches incur a $\mathbf{1 0 0} \times \mathbf{- 1 6 0 0} \times$ latency overhead (200-800 ms per request). Because of this significant delay overhead, LLM function calling is inappropriate for high-throughput and latencycritical backend services. However, selected 8B-class models (Llama 3.1 and Qwen 2.5) maintain 100% schema adherence under zero-temperature settings and reduce codebase size by approximately 33%, improving development agility. These findings demonstrate that LLM function calling is not a replacement for deterministic backend logic in latency-critical systems, but rather an effective orchestration layer for dynamic and natural language-driven workflows. We conclude by proposing a hybrid backend architecture that combines compiled logic for highperformance transactional paths with LLM-based reasoning for flexible interaction layers.
M. A. Ala’anzy, Adil Akhmetov, Amani Ibraheem et al.· 2026 6th International Confe...· 0 citations
LLM agents are increasingly used as transaction compilers: a user states an intent in natural language, and the model emits a structured object that an API can execute. JSON Schema and provider-level structured-output modes are useful because they remove a large class of parse failures, but they do not by themselves decide whether the object is a safe, faithful transaction. We introduce OrderBench, a deterministic benchmark for restaurant ordering agents that separates syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances. Across 2,400 Nebius Token Factory calls to four open models in prompt-only and JSON-schema modes, we find that schema-valid output can still have large semantic error rates. In the strongest model, both modes achieve 100% schema validity, yet semantic success remains near 80%; in weaker models, schema-valid unsafe acceptances occur in double digits. The result is a concrete engineering warning: structured output is a necessary interface layer, not a substitute for domain verification and fail-closed execution.
Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.
Tao Jin, P. Nguyen, Zhenzhu Yan et al.· 0 citations
ContainmentBench, a sandboxed benchmark comprising a 504-scenario specification dataset, a shared rollout-trace schema, and stage-scoped metrics for endpoint violations, logged propagation, and explicitly authorized taint-exposed proposals that commit, is introduced.