The Next Generation of Malware Defense using Deep Hashing and Neural Embeddings

Abstract

The proliferation of polymorphic and metamorphic malware has largely rendered traditional cryptographic signature-based detection ineffective, driving the adoption of similarity-based approaches. This survey systematically examines the evolution of binary similarity detection, tracing the trajectory from classical fuzzy hashing techniques—including ssdeep, sdhash, and TLSH—to contemporary deep representation learning architectures. We analyze state-of-the-art deep hashing methodologies, covering image-based representations via Convolutional Neural Networks (CNNs), structural control-flow graph modeling via Graph Neural Networks (GNNs), and assembly-level semantic analysis using Transformer architectures such as MalBERT and KEENHash. Furthermore, we critically assess the adversarial robustness of these embedding spaces across feature-space and problem-space threat models. By synthesizing recent theoretical developments and empirical benchmarks, this paper provides a unified taxonomy and outlines key challenges for resilient malware lineage tracking and zero-day threat detection.