Content-Hiding Attacks and Defenses for LLM-Based Agent-Skill Scanners

Abstract

AI agent “skills” (a natural-language SKILL.md plus bundled scripts) are distributed through marketplaces and public Git repositories, and a growing set of scanners classify them as safe or malicious, increasingly with a large language model (LLM) as the judge. We ask where these LLM-based scanners actually fail. Using a public benchmark of labeled skills and two production LLMs as scanners, we separate two attacker strategies: arguing (appending text that tries to talk the scanner out of its verdict) and hiding (positioning the malicious content so the scanner does not read it). Arguing is bounded and model-dependent: hand-written “this was approved by security” framing flips 18–24% of caught-malicious skills on a small model but only about 6–9% on a stronger one, and a feedback-driven adaptive attacker with a strong model does not beat the static version. Hiding is not: pushing the payload past the scanner’s read window flips 100% of caught-malicious skills on both models, and in-window dilution still flips 17–48%. Four real evasive skills built by an independent red team, and an independently built third-party scanner, show the same pattern. A whole-document chunked scan with repetition collapse recovers truncation hiding almost completely (100% → 11–14% on both models), but no defense we tried meaningfully reduces in-window dilution. The weakness of LLM skill scanners is perception, not reasoning.