When the Trigger Fails: Metamorphic Testing of Tool-Calling AI Agents

Abstract

Software agents built on foundation models depend on the prompt, the Trigger component, as their primary instruction interface, yet this component appears in roughly 1% of the test cases across the agent ecosystem. Recent literature calls for metamorphic testing of prompt behavior, but that call remains unanswered for the agentic case. We propose and evaluate a catalog of metamorphic relations over the Trigger, under the premise that transformations preserving the meaning of the prompt (paraphrase, typing noise, and language switching) should not alter the agent’s behavior, defined by the selected tool and its arguments; when they do, a silent behavioral regression is revealed. Across 1,600 controlled invocations of two open models run locally, over 40 tool-calling tasks with five runs each, violations prove prevalent and strongly concentrated in language switching (English→Portuguese), which alters behavior in up to 41% of the tasks against fewer than 11% for paraphrase and noise; the inconsistency falls chiefly on the arguments rather than on the selected tool; and the test discriminates between the two configurations evaluated (Wilcoxon, 𝑝 = 0.025), even after discounting an intrinsic flakiness affecting about 30% of the tasks. We release a reproducible framework with the relations, the data, and the analysis scripts, along with guidelines for integrating prompt regression testing into the development workflow.