Skip to content
Preprint

Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

Jul 2026 · 0 citations
Computer Science

TL;DR

It is demonstrated that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state.

Abstract

Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs). While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.

View source

Similar papers

Conference Open access 2026

LogSanitizer: Defending LLM-Integrated SOCs against Backdoor Triggers Delivered through Firewall Logs

LogSanitizer is proposed, a family of input sanitization defenses operating at two levels: a pre-prompt log-transformation pipeline that disrupts trigger patterns in the structured log representation, and a post-tokenizer perturbation strategy that corrupts trigger-bearing token configurations before they reach the model.

Leszek Wronski, Bogdan Ksiezopolski · 0 citations
Preprint Aug 2026

Breaking the Assumptions: Auditing Input-Side Jailbreak Defenses Against Semantic Attacks

Locally deployed Large Language Models (LLMs) via inference engines such as Ollama run without the moderation and abuse detection present in API-served models. Therefore, the safety of LLMs depends on the defense mechanisms used, and their effectiveness depends on the assumptions on which they were designed. This paper does an audit of defense mechanisms under jailbreak attacks on locally deployed models. Some defenses provide formal guarantees (SmoothLLM, Erase-and-Check, Sequential Monitors), while others rely on empirical detection results (Semantic Smoothing, Self-Denoised Smoothing, Perplexity Filtering). Instead of merely observing that defenses fail, we trace each failure back to the specific assumption: for every defense, we extract the condition it relies on, derive the empirical pattern a violation should produce, and test that prediction on six open-weight models (14B to 35B parameters) with a corpus of 100 jailbreak prompts taken from more than 40 public sources, totalling 13,800 evaluation records.

Aaditya Pratap, Harsh Kasyap, Somanath Tripathy · 0 citations
Jul 2026

Confused Deputy Attack Against Model Context Protocol

The model context protocol (MCP) has rapidly emerged as a standard framework for integrating large language models (LLMs) with external tools and resources. However, its metadata-driven and non-deterministic tool selection mechanism introduces a previously overlooked security threat. Leveraging this weakness, we uncover the confused deputy attack, where an adversarial server with subtly manipulated metadata covertly overshadows a benign one, intercepting tool invocations without exhibiting overtly malicious behavior. To systematically study this threat, we develop Puppet, the first automated security evaluation framework that: (i) enriches benign tool descriptions through selective requirement engineering to maximize semantic expressiveness, (ii) restructures them into LLM-preferred formats using description schema transformation, and (iii) applies name prioritization to introduce complementary lexical bias. Furthermore, Puppet synthesizes valid user queries to enable systematic attack evaluation. We comprehensively evaluate Puppet across 14 models from 6 providers on 2 MCP hosts, demonstrating tool selection hijacking rates up to 90.89% and end-to-end malicious payload execution rates up to 86.46%, while remaining undetectable by representative security scanners (MCP-Scan and McpSafetyScanner), which are architecturally incapable of detecting metadata-level manipulation attacks. Counterintuitively, we find that reasoning-enabled models are significantly more vulnerable than their non-reasoning counterparts. Our findings expose a critical design-level attack surface in the MCP ecosystem and highlight the urgent need for principled security safeguards.

Zhiyuan Li, Jingzheng Wu, Yuhao Peng et al. · 0 citations
Preprint Aug 2026

The Anatomy of a Prompt Injection: A Component Model for Structured Analysis

This paper formalizes the structure of prompt-injection artifacts, enabling defenders, red teamers, and cyber threat intelligence (CTI) teams to label, compare, and mutate attacks without relying on fragile string matching.

Jeremy McHugh · 0 citations
Preprint Aug 2026

A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

This work proposes a self-evolving test-time defense built around a persistent, cross-interaction rule memory that substantially reduces attack success rates while preserving benign utility, remains robust under an adaptive composite-wrapper attack, and does not increase over-refusal as the memory grows.

Tongshen Hu, Bryan Hooi · 0 citations