A manually constructed dataset of 150 English-language CTI reports, each represented as STIX 2.1 based graphs, provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution and indicates that locally deployed LLMs can support human reviewers in identifying annotation inconsistencies, but expert validation remains essential.
Abstract
Cyber threat intelligence (CTI) reports are typically written in unstructured formats, which complicates the extraction and analysis of important entities and adversarial behaviors. Although existing CTI research provides extraction tools, knowledge-graph frameworks, and MITRE ATT&CK mapped datasets, curated report-level datasets that preserve complex entity relationships and normalized adversarial behaviors remain limited. To address this limitation, this study presents a manually constructed dataset of 150 English-language CTI reports, each represented as STIX 2.1 based graphs, which includes 4,777 STIX entities, 5,817 STIX relationships in total, and 1,273 STIX attack-pattern entities (adversarial behaviors) mapped to 269 unique MITRE ATT&CK Enterprise techniques and sub-techniques. Twenty five randomly sampled reports were independently assessed by two cybersecurity researchers, which shows substantial inter-rater agreement. Disagreements were subsequently adjudicated to establish a gold-standard reference dataset. Four locally deployed open-source LLMs were evaluated as automated judges against this adjudicated reference sample. Qwen3.6:27B achieved the strongest overall performance, with a maximum kappa score of 0.803, micro-F1 scores exceeding 92%, and false-positive rates below 5%. The dataset provides a benchmark for CTI information extraction, knowledge-graph construction, incident analysis, and threat attribution. The findings further indicate that locally deployed LLMs can support human reviewers in identifying annotation inconsistencies, but expert validation remains essential.
Cyber threat intelligence (CTI) is foundational to modern cyber defense, yet much of it resides in unstructured reports whose volume and heterogeneity far exceed manual analysis, motivating research on automatically constructing knowledge graphs from CTI reports. However, existing approaches mainly extract partial information within a single report, leaving the cross-source setting unexplored, where the same threat is given unrelated names. Our key insight is that attack behaviors, once mapped to MITRE ATT&CK (a standardized catalog of attack techniques), can anchor the rest of a report. Attack behaviors are the adversarial actions a report describes, while contextual entities (e.g., threat actors, campaigns, and affected products) and Indicators of Compromise (IoCs; e.g., IP addresses) are their participants and traces. Attaching them to these anchors places every per-report graph in one canonical space. We realize this insight in BEACON, an LLM-driven framework for cross-source CTI knowledge graph construction. Its first stage extracts each report into a graph under a propose-then-verify paradigm, grounding candidates in report evidence and official ATT&CK definitions, to suppress LLM misclassification and hallucination. Its second stage merges these graphs with a hierarchical alignment strategy that applies signals in decreasing order of determinism, from character-level and semantic similarity to overlapping technique neighborhoods, iterating as merges pool neighborhoods. No existing benchmark links entities to technique anchors or provides cross-source alignment ground truth. We therefore construct and release two human-annotated datasets from 34 sources: to our knowledge the largest for report-level CTI extraction (8,395 elements) and the first for cross-source consolidation (3,487). On them, BEACON outperforms all baselines by at least 23% and 9%, respectively.
STINER, a taxonomy and expert-annotated corpus for extracting strategic intelligence from social media streams is introduced, and how social-media-driven extraction can surface early signals of the SafePay ransomware campaign prior to its retrospective characterization in vendor threat landscape reports is illustrated.
Yasir Ech-Chammakhy, Oussama Azrara, J. Chbili et al.· 0 citations
Accurately mapping attack behaviors described in Cyber Threat Intelligence (CTI) reports to the Tactics, Techniques, and Procedures (TTPs) of the MITRE ATT&CK framework is a critical challenge for responding to cyber threats and enhancing cyber resilience. However, traditional Large Language Model (LLM) and Retrieval-Augmented Generation (RAG) approaches face significant limitations. Specifically, the simple segmentation of unstructured CTI text leads to context fragmentation and the inclusion of extraneous descriptive details, which ultimately degrades the accuracy of TTP mapping. To solve these limitations, this paper proposes a methodology for constructing RAG queries based on structured fields. We extract attack behaviors from CTI reports as discrete events and organize them into seven fields (four required and three optional) representing the core components of the attack, which are designed to facilitate effective mapping to the MITRE ATT&CK framework. Our approach utilizes an LLM to perform event extraction and constructs optimized RAG queries through field combinations, thereby enhancing semantic alignment during the retrieval process. Experimental results demonstrate that the proposed method improves the F1-score by 0.249 compared to the baseline TTPFShot, achieving a maximum F1-score of 0.489 with the domain-specific model SecureBERT2. Moreover, while structured queries improve precision by constraining the retrieval scope to mitigate retrieval noise and reduce false positives, the effect of additional fields varies depending on the characteristics of the embedding model. Specifically, domain-specific models reach their peak performance with concise field configurations, whereas general-purpose models peak with the configuration integrating all fields. These results indicate that designing field combinations suited to the characteristics of the embedding model is essential, and that the proposed methodology provides a practical framework for high-precision TTP identification in complex CTI environments.
Seo-yeon Kim, Saewoom Lee, Seul-Ki Choi et al.· International Conference on...· 0 citations
CyberNER is introduced, a two-stage pipeline to solve the multi-type NER and alias canonicalization problem in APT CTI reports and achieves a Macro-F1 score of 0.853, outperforming all four baselines.
Unnamalai K, Suriakala M· International journal of com...· 0 citations
Cybercrime investigations increasingly depend on the ability to interpret large volumes of automated security events. For organizations without dedicated security operations centres, a situation common among small and medium enterprises, the manual translation of raw alerts into structured threat intelligence represents a critical bottleneck that slows investigative triage and limits cross-case comparability. This paper evaluates an automated enrichment pipeline designed to address this bottleneck by mapping security events to standardised adversary behaviour labels drawn from the MITRE ATT&CK framework, supporting both operational response and cybercrime investigation workflows. We compare three pipeline configurations, a general-purpose encoder model, a cybersecurity domain-adapted variant, and a hybrid system incorporating a metadata-aware re-ranker, across 1,566 alerts from a controlled nine-stage adversarial experiment with 471 ground-truth labelled alerts spanning 12 adversary techniques, and 8,465 Windows events from an external dataset. Domain adaptation proves to be a prerequisite for operational viability where the general-purpose model assigns meaningful similarity scores to fewer than 1% of alerts, while the fine-tuned model does so for all 1,566. The hybrid system achieves 88.54% micro Top-1 accuracy and a class-balanced macro-F1 of 0.244 against a majority-class predictor of 0.182, confirming genuine multi-class benefit beyond what a trivial always-predict strategy provides. The metadata re-ranker resolves a specific ambiguity between valid-accounts and account-removal predictions, correcting 400 misattributions with zero adverse effects and 98.3% precision, a statistically significant improvement stable across all tested parameter settings.
E. Sherif· International Journal of Cyb...· 0 citations
Automated extraction of structured threat information from unstructured cyber threat intelligence (CTI) underpins modern security operations, yet the supporting machine learning resources are almost exclusively English: no annotated Arabic CTI named entity recognition (NER) corpus has been published. We introduce AraCTI-NER, a dataset of 10,312 token-level annotated samples (275,530 tokens; 42,360 entity spans) over eight STIX-inspired entity types, built by an LLM-assisted pipeline seeded with authentic Arabic cybersecurity articles, structurally validated and rebalanced through targeted generation. We benchmark seven encoders from three families (Arabic-specialized, English cybersecurity-adapted, and multilingual) over three seeds under strict entity-level metrics, and release a 408-sentence expert-audited test subset (ATS-gold) whose reliability is quantified by a second independent expert validation (inter-annotator agreement 0.878 entity-level F1). XLM-RoBERTa Large attains the best mean F1 (0.7603; 0.7674 on ATS-gold), with AraBERTv2 close behind (0.7491), while both English-only cybersecurity encoders fall to ≈0.63, a separation that holds across every seed and survives expert correction, with the ≈3-point F1 decrease from ATS-silver to ATS-gold concentrated in Vulnerability and TTP. On 350 doubly annotated sentences from authentic Arabic cyber-incident news, a shift in both provenance and register, the strongest model reaches F1 = 0.5429 against an inter-annotator F1 of 0.616. AraCTI-NER establishes the first reproducible baseline for Arabic CTI NER and identifies domain-adaptive Arabic cybersecurity pre-training as the highest-value next step.
