TFC-MAL: APPLICATION OF TRIPLET LOSS FOR MALWARE DETECTION BASED ON IMAGE REPRESENTATION

Abstract

Detecting polymorphic and zero-day malware remains a significant challenge due to the high similarity in characteristics between malicious variants and benign files, which reduces the effectiveness of traditional Cross-Entropy-based approaches. This paper proposes the TFC-MAL (Triplet-based Feature Compactness for Malware Detection) framework, which employs Triplet Loss to learn a discriminative embedding space from malware image representations. By optimizing relative distances with a fixed margin, Triplet Loss generates embedding vectors with high intra-class compactness and clear inter-class separation, thereby enhancing generalization to previously unseen samples. Experimental results demonstrate that TFC- MAL combined with an SVM classifier achieves the highest accuracy of 90.83% on the Unknown-ClamAV dataset, representing an improvement of 11.43% over the current state- of-the-art method. In addition, the proposed system maintains high performance on benign samples (96.76%) and known malware samples (96.84%) when using an MLP classifier. These results confirm that Triplet Loss provides superior generalization capability for detecting zero-day malware and polymorphic variants. Keywords: Malware detection; image-based malware representation; deep learning; Triplet Loss; zero-day malware.