AttackLogGen: Benchmarking LLMs for generating attack logs

Abstract

Multi-Host Multi-Stage (MHMS) cyberattacks are increasingly sophisticated, spanning multiple systems through coordinated stages and evading traditional single-host detection approaches. Such attacks leave correlated traces across diverse log sources, including system, audit, and authentication logs. However, research on MHMS intrusion detection is constrained by the scarcity of realistic multi-host log datasets, as real-world data are difficult to share due to sensitivity and the operational complexity of reproducing attacks in controlled environments. To address this challenge, we present AttackLogGen, a framework for benchmarking Large Language Models (LLMs) in generating realistic MHMS logs. Malicious activities are modeled using the MITRE ATT&CK knowledge base, while benign background behavior is synthesized from generic event patterns across interconnected hosts. Leveraging LLM contextual reasoning, AttackLogGen produces logs tat preserve causal dependencies, temporal ordering, and cross-host interactions characteristic of complex attacks. The framework integrates coordinated modules to generate time-aligned and semantically coherent system behavior. For validation, we built a small-scale MHMS testbed executing end-to-end multi-stage attacks to collect ground-truth logs, complemented by single-system data from the LogHub dataset. Evaluation across semantic similarity, distributional consistency, discriminability tests, and structural fidelity shows that LLM-generated logs—particularly using GPT-4o-mini, closely approximate real MHMS activity, enabling safe and scalable experimentation for intrusion-detection research.