Quantum Support Vector Machine for Encrypted Botnet Command and Control Traffic Detection Without Payload Inspection

Abstract

Modern botnets use encrypted Command-and-Control (C2) communications, which poses a major challenge for network security given the fact that payloads are encrypted, which makes the use of traditional signature-based inspection methods ineffective. In this work, we introduce QSVM Shield, a hybrid quantum-classical solution for encrypted botnet C2 traffic detection purely based on network flow metadata following a Quantum Support Vector Machine (QSVM) algorithm. The proposed technique does not need to decrypt the packet payloads to detect threats, but instead uses statistical characteristics of encrypted packet flows to achieve this goal while maintaining the privacy of the communications. First, the scale variations in the essential flow attributes are reduced and feature consistency is enhanced by applying logarithmic feature transformation and MinMax normalization. Principal Component Analysis (PCA) is then used to find a low dimensional representation that is compact enough for quantum encoding. The resulting reduced feature vectors are then loaded into a 4-qubit quantum feature map that is implemented using PennyLane, with parameterized quantum rotations and entanglement operations building up a nonlinear quantum kernel used to model complex relationships between the various encrypted traffic samples. The generated kernel matrix is used with a classical Support Vector Classifier that has a precomputed kernel to classify benign traffic from encrypted C2 communication of botnets. For a practical security operation, all the framework is deployed as a Flask based web application, where it enables single instance prediction, batch CSV analysis, performance visualization and scan history management. Experimental results with network flow data encrypted under the proposed hybrid quantum-classical pipeline are presented and show that the proposed hybrid quantum-classical pipeline can achieve effective detection and discrimination of malicious C2 traffic while keeping the inference process lightweight, using only flow-level information. The combination of quantum kernel learning and privacy-preserving encrypted traffic analysis creates a technically secure and practically viable solution to next generation intelligent network intrusion detection.