ALTR: Adaptive Defense Framework Against Adversarial LLM Threats in Critical-Domain Applications

Abstract

The fast deployment of large language models (LLMs) in critical-domain applications such as financial systems has introduced new types of security vulnerabilities (e.g., adversarial prompt injections, semantic obfuscation, and multi-stage social engineering attacks) that conventional countermeasures cannot typically handle. These vulnerabilities need to be detected through the natural language interface interactions, and relevant security policies need to be applied based on the context of the interactions. In this paper, we present a novel Adaptive LLM Threat Response (ALTR) framework, that supports trustworthy decision-making in LLM-enabled applications by integrating real-time behavioral anomaly detection with context-aware prompt validation to mitigate the risks of vulnerable LLMs. Specifically, we develop a Behavioral Anomaly Engine (BAE) that uses Isolation Forests to identify anomalies in user behavior during LLM interactions in financial systems. In addition, we develop a Context-Aware Prompt Classifier (CAPC) that employs DistilBERT and syntactic analysis to determine malicious inputs. To address the latency requirements in threat response, we create a Threat Response Memory (TRM) knowledge base that stores the aggregated behavioral and semantic risk signals from prior user sessions, and uses LSTM to perform longitudinal threat analysis to predict threat severity. We perform validation experiments using public datasets and a taxonomy of AI-based threats outlined in the Cisco AI Defense Capture-the-Flag (CTF) challenge. Our experiment results show the effectiveness of our ALTR approach by achieving an accuracy of 0.961, an AUC of 0.961, and a low false-negative rate of 0.9%, while maintaining an inference latency ($<\mathbf{2 0 m s}$) of across a range of LLM security vulnerabilities.