Experimental Analysis of Adversarial Attacks on Agentic AI Systems with Layered Defense Mechanisms

Abstract

Agentic AI architectures leverage LLMs, tool execution, and decision-making mechanisms to handle a variety of tasks that go beyond text generation. The ability of these agents to accomplish such non-textual tasks poses new security concerns, particularly against adversarial prompting attacks. In this paper, we present a reproducible setup to investigate prompt-injection-like attacks on an agentic system locally, with the option to turn on and off the safety pipeline to facilitate comparative studies. Our architecture implements a keyword-based filtering approach to prompt injection, risk-weighting, and policy-based gating mechanisms for critical tools. We evaluate the proposed framework using a dataset of 108 prompts, including 104 adversarial and 4 benign prompts. Experimental results show that enabling the layered defense reduces the attack success rate from 72.5% to 18.3% while maintaining acceptable usability. Although effective against explicit attacks, the framework remains vulnerable to semantic paraphrasing and context-aware adversarial prompts, highlighting the need for semantic defense mechanisms.