Android Malware Detection Using a Hybrid XGBoost–Random Forest Model with Cost-Sensitive Learning

Abstract

Android apps have more users and consequently are more susceptible to smart malware, raising significant privacy, data protection, and security concerns. While machine-learning approaches to malware detection have yielded good results, most existing approaches optimize symmetric evaluation metrics and assume that all types of misclassification errors are equal. But in a real environment of Android security, false negatives are a far more serious danger than false positives, as malicious apps are more likely to be mistaken for being benign. In order to alleviate these issues, the present study introduces a cost-sensitive malware detection framework while at the same time incorporating Extreme Gradient Boosting (XGBoost) and Random Forest (RF) classifiers with asymmetric misclassification costs. A feature selection process is used to select only the most discriminative features, which is done using a combination of impurity-based ranking, recursive feature elimination, and permutation importance, to achieve both efficiency and interpretability The proposed cost-sensitive hybrid XGB–RF model with a cost ratio of 5:1 false-negative to false-positive correctly reduces the number of false negatives by 54.4% and increases recall by 2.2 percentage points compared with the cost-insensitive standard hybrid model, with a marginal accuracy loss of 0.4%. The results have demonstrated that the proposed cost-sensitive optimization of hybrid ensembles significantly enhances the operational reliability of the system for detecting Android malware while maintaining overall classification accuracy, making it an intuitive, explainable, and deployable solution instead of deep learning-based solutions.