TL;DR
Experimental results show that LLMs, when guided by rubric-based prompts and supplemented with ATT&CK domain knowledge, achieve robust performance across detection, localization, and TTP mapping tasks.
This paper presents a framework for evaluating prompt injection attacks against LLM-based log interpretation using log traces generated during real cyber attacks, and creates adversarial examples through generic injection generation, refinement, and attack-specific optimization.
Multi-Host Multi-Stage (MHMS) cyberattacks are increasingly sophisticated, spanning multiple systems through coordinated stages and evading traditional single-host detection approaches. Such attacks leave correlated traces across diverse log sources, including system, audit, and authentication logs. However, research on MHMS intrusion detection is constrained by the scarcity of realistic multi-host log datasets, as real-world data are difficult to share due to sensitivity and the operational complexity of reproducing attacks in controlled environments. To address this challenge, we present AttackLogGen, a framework for benchmarking Large Language Models (LLMs) in generating realistic MHMS logs. Malicious activities are modeled using the MITRE ATT&CK knowledge base, while benign background behavior is synthesized from generic event patterns across interconnected hosts. Leveraging LLM contextual reasoning, AttackLogGen produces logs tat preserve causal dependencies, temporal ordering, and cross-host interactions characteristic of complex attacks. The framework integrates coordinated modules to generate time-aligned and semantically coherent system behavior. For validation, we built a small-scale MHMS testbed executing end-to-end multi-stage attacks to collect ground-truth logs, complemented by single-system data from the LogHub dataset. Evaluation across semantic similarity, distributional consistency, discriminability tests, and structural fidelity shows that LLM-generated logs—particularly using GPT-4o-mini, closely approximate real MHMS activity, enabling safe and scalable experimentation for intrusion-detection research.
Large language models (LLMs) embedded in enterprise workflows cannot structurally distinguish legitimate instructions from adversarial ones in the same token stream, making prompt injection OWASP's top LLM risk for two consecutive editions a persistent threat across direct and indirect vectors. This paper presents PromptShield-RT, a layered, real-time, model-agnostic framework combining input normalization and provenance tagging, lexical-heuristic pattern matching, a statistical classifier, structural anomaly features, and calibrated risk fusion, with policy-driven mitigation (allow/sanitize/quarantine/block) and an explainable, adaptive-feedback mechanism for SOC workflows. We construct an original evaluation corpus, SynPI-Bench (n = 450, six categories), and a template-disjoint held-out generalization set (n = 31) with novel phrasings, obfuscation encodings, and adversarial hard-negative benign text. Using template-grouped 5-fold cross-validation, the fused pipeline achieves 92.4% accuracy (F1 = 0.930, AUC = 0.990), outperforming heuristic-only (57.0%) and naive-averaged (59.2%) baselines, while a lexical classifier reaches 85.9% with lower precision. We report a pronounced generalization gap on the held-out set (48.4% accuracy, 90% false-positive rate on hard negatives), quantifying a known limitation of surface-lexical defenses. The pipeline achieves sub-millisecond P95 latency (0.266 ms), within typical 50 ms enterprise SLAs. We situate PromptShield-RT relative to structural, architectural, and guardrail-product defenses, arguing for layered, defense-in-depth architectures, with reproducible code provided.
The reasoning model DeepSeek-R1 was more susceptible to jailbreak attacks than the non-reasoning model DeepSeek-V3, and different prompt-level defenses were effective against static jailbreaks.
We propose a two-stage large language model (LLM) framework for zero-shot detection of insider threats and advanced persistent threats (APTs) from heterogeneous security logs. The framework models user activity as chronological timelines and incorporates retrieval-augmented generation (RAG) to provide personalised behavioural context from each user's historical activity. Rather than performing end-to-end classification directly from raw logs, it first generates structured, interpretable sets of threat-specific risk indicators, which are then classified jointly across temporal sequences to capture attack patterns spanning multiple windows.The framework is evaluated on two benchmark datasets, CERT r5.2 for insider threat detection and PicoDomain for APT detection, using four combinations of two open-weight LLMs under both retrieval and non-retrieval settings. All configurations outperform the previous state-of-the-art LLM-based framework (GABM), with the best configuration improving the F1-score by 11.40 percentage points on CERT r5.2 and 31.50 percentage points on PicoDomain. Results further show that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context. The most effective assignment of LLMs to the two stages depends on the dataset. These findings show that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance.
DeCNIP (Defense with Critical Neuron Isolation Pruning), which leverages representational analysis to identify and neutralize backdoors in a unified pipeline, is introduced, which achieves over 95% relative reduction in Attack Success Rate (ASR), outperforming seven state-of-the-art defenses with only 0.1% neuron intervention.