TrafficBASS: A Boundary-Aware Active Learning Framework for Endpoint Advanced Persistent Threat Detection

Abstract

Advanced Persistent Threats (APTs) pose significant challenges to endpoint intrusion detection systems (IDSs) because their traffic often exhibits long-tail distributions, temporal evolution, boundary ambiguity, and concept drift. Endpoint deployment further suffers from scarce labeled samples and limited computational resources. To address these challenges, we propose TrafficBASS, https://github.com/AnonymousCodeFiles/TrafficBASS a boundary-aware pool-based active learning framework for endpoint APT traffic detection. TrafficBASS coordinates three components: (i) a Learnable Feature Projection (LFP) module that combines random-forest-selected statistical flow features and packet-length sequence features into a session-level representation; (ii) a Boundary-Aware Sampling Strategy (BASS) that selects high-value unlabeled sessions by considering predictive uncertainty, class-balancing priority, and feature-space diversity near the decision boundary, while using an elastic memory bank and bounded feature-space adversarial expansion to preserve rare ambiguous regions; and (iii) an Adaptively Lightweight Fine-tuning (ALF) strategy that adapts an ALBERT-based encoder with LoRA to reduce trainable parameters for endpoint-constrained updates. Experiments on public traffic datasets show that TrafficBASS improves APT traffic detection under limited labeling budgets, long-tail distributions, and concept drift compared with representative active learning strategies and traffic Transformer baselines. Additional ablation, robustness, case-study, and efficiency analyses further characterize the contribution and limitations of each component.