Deep learning-based android malware detection under realistic evaluation constraints: an architecture-centric review and reliability-aware empirical validation

Abstract

Deep learning (DL) has reshaped Android malware detection (AMD) by enabling models to learn from heterogeneous Android artifacts, including permissions, DEX and bytecode representations, API and system-call traces, network traffic, image encodings, and program graphs. However, high benchmark accuracy does not necessarily imply reliable, robust, explainable, reproducible, or deployable malware detection. This study examines DL-based AMD under realistic evaluation constraints through an architecture-centric systematic review and reliability-aware empirical validation. A PRISMA-guided review of studies published between 2021 and 2026 was conducted to analyze architecture families, artifact representations, dataset transparency, validation realism, robustness and evasion evidence, explainability, reproducibility, and deployment readiness. Controlled empirical validation was also conducted on CICMalDroid 2020 using representative CNN, GRU, Transformer, and Hybrid CNN–LSTM models. The review shows that DL-based AMD is architecturally diverse, but its supporting evidence remains incomplete. Most studies report datasets, feature sources, model architectures, and conventional metrics, whereas robustness testing, artifact-grounded explainability, reproducibility assets, and deployment evidence remain inconsistent. The empirical validation shows that although all models achieve strong clean cross-validation performance, class-wise analysis, malware false-negative behavior, repeated-run stability, and multi-level test-time feature masking reveal architecture-dependent reliability differences that are not captured by aggregate accuracy. The study contributes an architecture–artifact–validation triad and a claim-aligned reliability audit to align AMD claims with the maturity of supporting evidence. Overall, the findings support a transition from accuracy-centric benchmarking to reliability-centric evaluation, where models are assessed using credible datasets, realistic validation, robustness, interpretability, reproducibility, and operational feasibility.