A systematic review and structured descriptive synthesis of research on defenses against prompt-based attacks in language model and agent systems reveals trade-offs between security effectiveness, performance, and system complexity as well as major gaps in benchmarks, indirect attack coverage, and multi-agent evaluation.
Abstract
Current language model deployments face growing security challenges from prompt-based attacks, including jailbreaks, direct and indirect prompt injection, and instruction hijacking, which often evade traditional rule-based safeguards. As these models are increasingly integrated into agent-based systems, retrieval pipelines, and tool-driven workflows, such attacks exploit their natural language interfaces to bypass safety constraints and manipulate system behavior, in some cases leading to data leakage or unauthorized actions. Recent research has proposed a wide range of defense mechanisms ranging from prompt-level filtering and model-level detection to pipeline wrappers and multi-agent protection frameworks. Many of these methods report strong results in controlled experiments, yet their effectiveness depends heavily on the assumed threat model, the datasets used, and the evaluation protocol. This paper presents a systematic review and structured descriptive synthesis of research on defenses against prompt-based attacks in language model and agent systems. By integrating and comparing findings across multiple studies, we identify major attack categories, commonly adopted defense strategies, deployment stages, and evaluation trends, while also highlighting limitations related to generalization, robustness, and real-world applicability. The analysis reveals trade-offs between security effectiveness, performance, and system complexity as well as major gaps in benchmarks, indirect attack coverage, and multi-agent evaluation. The systematic review concludes by outlining future research priorities, including pipeline-aware defense design, adaptive and layered protection mechanisms, and more realistic evaluation practices to support the development of robust and deployable prompt security solutions.
Large Language Model (LLM)-based agents automate complex tasks by integrating external tools such as web browsers, e-mail clients, file readers, and APIs, but this same integration exposes them to indirect prompt injection (IPI) attacks, in which malicious instructions hidden in tool content hijack the agent. A central but often overlooked question is how defending against such attacks affects the LLM and its own task performance and computational efficiency. In this study, we design a comprehensive testbed and a layered defense, Spotlight-Guard, that combines spotlighting-based input isolation, an LLM detection-and-quarantine pipeline, and instruction integrity based on a Hash-based Message Authentication Code (HMAC) into a single framework, and we evaluate it jointly along two axes: security and LLM performance. Experiments on locally hosted 7B-class open-weight models (Qwen-2.5-7B, Mistral-7B, and DeepSeek-Coder) use Attack Success Rate (ASR) for security and benign-task success rate together with confusion-matrix-based metrics (precision, recall, and F1) for task performance, all with bootstrap 95% confidence intervals. Across a stratified, fixed-seed benchmark of 250 adversarial and 250 benign cases per configuration, the full system reduces the ASR from 36.0% to 17.2% while preserving a 97.2% benign-task success rate and raising the detection F1 from 0.749 to 0.892, demonstrating that strong protection need not degrade the model’s task performance. A component ablation isolates each layer’s contribution, an adaptive-attack evaluation confirms a low ASR (6.7%) under attacks crafted to target the pipeline, and an analysis of computational cost (model invocations per request) quantifies the efficiency overhead, characterizing the security–performance trade-off of layered defenses on open-weight LLMs.
Doygun Demirol, Murat Aydoğan· Applied Sciences· 0 citations
The reasoning model DeepSeek-R1 was more susceptible to jailbreak attacks than the non-reasoning model DeepSeek-V3, and different prompt-level defenses were effective against static jailbreaks.
V. T. Hayashi, Milton Pedro Pagliuso Neto, C. Miers et al.· IEEE Access· 0 citations
Large language models (LLMs) embedded in enterprise workflows cannot structurally distinguish legitimate instructions from adversarial ones in the same token stream, making prompt injection OWASP's top LLM risk for two consecutive editions a persistent threat across direct and indirect vectors. This paper presents PromptShield-RT, a layered, real-time, model-agnostic framework combining input normalization and provenance tagging, lexical-heuristic pattern matching, a statistical classifier, structural anomaly features, and calibrated risk fusion, with policy-driven mitigation (allow/sanitize/quarantine/block) and an explainable, adaptive-feedback mechanism for SOC workflows. We construct an original evaluation corpus, SynPI-Bench (n = 450, six categories), and a template-disjoint held-out generalization set (n = 31) with novel phrasings, obfuscation encodings, and adversarial hard-negative benign text. Using template-grouped 5-fold cross-validation, the fused pipeline achieves 92.4% accuracy (F1 = 0.930, AUC = 0.990), outperforming heuristic-only (57.0%) and naive-averaged (59.2%) baselines, while a lexical classifier reaches 85.9% with lower precision. We report a pronounced generalization gap on the held-out set (48.4% accuracy, 90% false-positive rate on hard negatives), quantifying a known limitation of surface-lexical defenses. The pipeline achieves sub-millisecond P95 latency (0.266 ms), within typical 50 ms enterprise SLAs. We situate PromptShield-RT relative to structural, architectural, and guardrail-product defenses, arguing for layered, defense-in-depth architectures, with reproducible code provided.
Fatimah Alhamzawi· Al-Noor Journal of Engineeri...· 0 citations
This paper proposes SecureMCP, a policy-enforced framework that integrates Role-Based Access Control with an MCP server to establish multi-layer defense for LLM-generated SQL execution, and evaluates filter performance—false positive rate (FPR) and false negative rate (FNR))—separately from LLM generation quality.
Wonbae Kim, Hee-Kyong Yoo, Nammee Moon· Applied Sciences· 0 citations
A four-layer taxonomy mapping 13 vulnerability types across perception, brain, action, and interaction layers is contributed, and seven open problems centered on containment are identified.
Md Jafrin Hossain, Mohammad Arif Hossain, Nirwan Ansari· 0 citations
Prompt injection poses a significant security risk to Retrieval-Augmented Generation (RAG) systems, enabling adversaries to embed malicious instructions within retrieved documents and hijack model behavior to exfiltrate sensitive information or execute unauthorized actions. This work presents a modular dynamic evaluation environment for systematically testing and comparing defense mechanisms against prompt injection attacks in RAG architectures. The framework simulates diverse injection scenarios targeting the retrieval pipeline, integrates optional mitigation strategies such as input filtering, prompt rewriting, and retrieval-aware defenses, and automatically logs model behavior to assess attack success. By varying retrieval parameters and quantifying defense robustness across diverse attack verticals and RAG configurations, the system enables reproducible and scalable evaluation of prompt injection resilience. The results highlight strengths and weaknesses of existing defenses in RAGspecific threat models and establish a foundation for standardized benchmarking of defense mechanisms in knowledge-grounded generative AI systems.
Ankita John, Geethapriya Thamilarasu· International Conference on...· 0 citations