A tool-using LLM agent that reads attacker-controlled web content while holding a secret faces indirect prompt injection: the content may make it exfiltrate the secret, which may make it exfiltrate the secret.
Abstract
A tool-using LLM agent that reads attacker-controlled web content while holding a secret faces indirect prompt injection: the content may make it exfiltrate the secret. In a safe synthetic lab (canary secret, mock tools, matched clean-vs-poisoned metric) we report the framing gap: across six models, ten overt injection classes are refused (gpt-4o 0%), but reframing the identical leak as a mandatory integrity signature, config field, or look-alike"trusted"host drives gpt-4o 0% to 100%. The attack is cheap, and its cost is three-level: paraphrasing a known mechanism is trivial (96% at 3 wordings), swapping the field inside a known-effective template is also cheap (up to 60%), while authoring a fresh page around a new mechanism is hard (0/130) -- the reusable asset is the template, not the mechanism. An ablation shows the mechanism is instruction/data confusion, not defeated alignment: removing the confidentiality policy leaves base attacks at 0% and moves reframing only 31.9% to 38.1%. What closes the gap is payload-blind checks: a destination allow-list (0%, when destinations are closed) and a capability-isolating planner/reader split (0%). A broad"in any form"policy clause also closes it at the acting model (to 0%) but is brittle (dropping the catch-all reopens it to 48.8%). A published fine-tuning defense (SecAlign, CCS 2025) does not close it on a tool agent (32.5%, positive-control-validated), nor does channel separation (38.8%); an output-normalizing guard loses to a held-out encoding (ROT13, 100%). Robustness comes from constraining the destination or isolating the capability, not from the acting model recognizing the attack.
The explosive prompt is introduced, a conditional payload that stays dormant until an attacker-chosen trigger is met, in effect a training-free, inference-time backdoor planted in a single piece of retrieved content.
Justin Szczepaniak, Elad Feldman, Naum Viner et al.· 0 citations
The findings show that defense evaluation should report attack outcomes, goal feasibility, legitimate-task utility, action availability, operational measures, and run-to-run variation.
Adil Khan, Khaled AlKhanbashi, Azza Mohamed· Computers· 2 citations
It is argued that evaluation validity is a prerequisite for, not a footnote to, defense claims in agentic security, and a harness is provided to provide an instrument that enforces it.
This work assesses indirect prompt injection in DeepSeek Harness, using AI-Infra-Guard (A.I.G) to construct tests, deliver controlled taint, execute DSH, collect traces, and judge outcomes, and relates these results to DSH's treatment of tool results, additional contexts, and tool-call policy hooks.
Zong-Hao Ying, Xiang-Fan Wu, Hui-Yu Wu et al.· 2 citations
A framework to measure the trust profile of an arbitrary LLM based on a variety of payload framings sent through different channels is presented, and cross-channel fragmentation attacks that distribute seemingly benign payloads across two or three channels are devised.
Indirect prompt injection embeds malicious instructions within external content retrieved by LLM-based agents, altering target behavior without user authorization. We introduce pikit, a research toolkit designed to systematically evaluate these threats across three core dimensions: attacks (13 methods), channels (16 ca...
Zong-Hao Ying, Xiang-Fan Wu, Bo Yang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.