ExE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison, and highlights how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance.
Abstract
Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.
This work presents a two-phase evaluation of ten Llama variants using the OWASP Top 10 for LLM Applications, and applies nine encoding obfuscations to the same prompts, which fully bypasses all text-only models.
Nourin Shahin, I. Alsmadi· Practice and Experience in A...· 0 citations
To determine the real-world effectiveness of machine learning based malware detection, it is vital to evaluate its robustness against highly capable adversaries. However, state-of-the-art attacks do not effectively model realistic adversaries, as they often assume access to privileged information such as the training data, feature space, or confidence scores of the target. In this work, we present Replicant, a deep reinforcement learning framework that learns the realistic task of evasion under a strict label-only black-box threat model. Replicant learns a reusable policy on how to modify a malware sample and when to query the target, which transfers across samples, detectors, and feature spaces. Across seven Android malware detectors and three feature spaces, Replicant is the strongest and most query-efficient approach achieving a mean attack success rate of 78.8%, a relative improvement of 20.9%-39.2% over the state-of-the-art. Furthermore, when used for adversarial training, Replicant also outperforms the state-of-the art by producing detectors with more generalizable robustness. With Replicant we demonstrate that learning the task of evasion not only results in stronger attack performance but, crucially, provides a better signal for hardening malware detectors.
Shae McFadden, Ilias Tsingenopoulos, Mario D'Onghia et al.· 0 citations
Modern malware frameworks use advanced evasion techniques that bypass traditional detection methods; thus entail advanced analytical frameworks for comprehensive and robust analysis. This study provides a comparative analysis of several frameworks that utilize Explainable Artificial Intelligence (XAI), Generative Adversarial Networks (GANs), and Large Language Models (LLMs) to provide a dynamic approach to malware behaviour. The review consisted of five key metrics to assess these three frameworks: detection performance, explainability, robustness against adversarial attacks, behavioral interpretation, and automated reporting capabilities. The results indicate that Deep Learning models have attained high accuracy in detect-ing and identifying malicious code, but are not interpretable. The GAN-based frameworks are highly effective in generating adversarial samples for robustness testing. Conversely, LLM-based approaches are highly effective for generating automated forensic reports, but are not yet fully integrated into malware detection workflows. The analysis highlights a research gap pertaining to the lack of integrated frameworks for adversarial analysis, explainability and automated forensic reporting. This study proposes a unified approach of malware analysis incorpo-rating XAI, GAN and LLM to enable the development of more effective malware detection tools with more transparency, deeper analytical insight and advanced forensic decision-making.
A. Verma, Neha Gupta, Akash Saxena et al.· International Journal of Inn...· 0 citations
A deployment-minded pipeline that integrates JSON-aware canonicalization that standardizes benign representational variance while preserving attack-relevant semantics, with the largest reductions on JSON wrapping and on multi-step chains such as homoglyph ?
Ali Taqi, Hassan Hassan, Ghaidaa Abdulsaheb· Journal of Applied Science a...· 0 citations
In the last years, several gradient-based attacks have been developed to disclose vulnerabilities of deep neural models and strengthen evasion and transferability abilities of adversarial examples created in various domains. In Windows PE malware detection, various gradient-based methods have been recently tested to optimize the editing or injection of adversarial bytes in unused file sections, and create adversarial malicious PE files to evade deep neural models developed for malware detection. On the other hand, gradient information is commonly used in eXplainable AI (XAI) to obtain explanations for opaque decision-making processes of deep neural black-boxes, while Adversarial XAI has recently emerged as an Adversarial Learning field to investigate the security landscape of XAI. In this study, we describe a holistic, gradient-based attack method, named GAME4EXE, formulated to optimize editing of adversarial bytes in the DOS Header and DOS Stub sections of Windows PE malware. The optimization is driven by the dual adversary purpose of generating adversarial malware that evades a target deep neural model, and aligning the explanation of the evading decisions with that of a goodware-like explanation. A preliminary evaluation shows that a gradient-based attack can be effectively formulated in the Windows PE malware domain, to equip the evasion of a deep neural model with the capability to coherently deceive its decision process explainer and harden attack detection.
Luca Lobascio, Giuseppina Andresini, A. Appice et al.· 2026 IEEE European Symposium...· 0 citations
This paper presents a framework for evaluating prompt injection attacks against LLM-based log interpretation using log traces generated during real cyber attacks, and creates adversarial examples through generic injection generation, refinement, and attack-specific optimization.
Max Landauer, Florian Skopik, Markus Wurzenberger et al.· 0 citations