It is demonstrated that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state.
Abstract
Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.
LogSanitizer is proposed, a family of input sanitization defenses operating at two levels: a pre-prompt log-transformation pipeline that disrupts trigger patterns in the structured log representation, and a post-tokenizer perturbation strategy that corrupts trigger-bearing token configurations before they reach the model.
Leszek Wronski, Bogdan Ksiezopolski· Proceedings of the 23rd Inte...· 0 citations
Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-served models. Therefore, the safety of LLMs depends on the defense mechanisms used, and their effectiveness depends on the assumptions on which they were designed. This paper does an audit of defense mechanisms under jailbreak attacks on locally deployed models. Some defenses provide formal guarantees (SmoothLLM, Erase-and-Check, Sequential Monitors), while others rely on empirical detection results (Semantic Smoothing, Self-Denoised Smoothing, Perplexity Filtering). Instead of merely observing that defenses fail, we trace each failure back to the specific assumption: for every defense, we extract the condition it relies on, derive the empirical pattern a violation should produce, and test that prediction on six open-weight models (14B to 35B parameters) with a corpus of 100 jailbreak prompts taken from more than 40 public sources, totalling 13,800 evaluation records.
The model context protocol (MCP) has rapidly emerged as a standard framework for integrating large language models (LLMs) with external tools and resources. However, its metadata-driven and non-deterministic tool selection mechanism introduces a previously overlooked security threat. Leveraging this weakness, we uncover the confused deputy attack, where an adversarial server with subtly manipulated metadata covertly overshadows a benign one, intercepting tool invocations without exhibiting overtly malicious behavior. To systematically study this threat, we develop Puppet, the first automated security evaluation framework that: (i) enriches benign tool descriptions through selective requirement engineering to maximize semantic expressiveness, (ii) restructures them into LLM-preferred formats using description schema transformation, and (iii) applies name prioritization to introduce complementary lexical bias. Furthermore, Puppet synthesizes valid user queries to enable systematic attack evaluation. We comprehensively evaluate Puppet across 14 models from 6 providers on 2 MCP hosts, demonstrating tool selection hijacking rates up to 90.89% and end-to-end malicious payload execution rates up to 86.46%, while remaining undetectable by representative security scanners (MCP-Scan and McpSafetyScanner), which are architecturally incapable of detecting metadata-level manipulation attacks. Counterintuitively, we find that reasoning-enabled models are significantly more vulnerable than their non-reasoning counterparts. Our findings expose a critical design-level attack surface in the MCP ecosystem and highlight the urgent need for principled security safeguards.
Zhiyuan Li, Jingzheng Wu, Yuhao Peng et al.· ACM Transactions on Software...· 0 citations
This paper formalizes the structure of prompt-injection artifacts, enabling defenders, red teamers, and cyber threat intelligence (CTI) teams to label, compare, and mutate attacks without relying on fragile string matching.
This work proposes a self-evolving test-time defense built around a persistent, cross-interaction rule memory that substantially reduces attack success rates while preserving benign utility, remains robust under an adaptive composite-wrapper attack, and does not increase over-refusal as the memory grows.