A Novel Entropy-Based Framework for Hybrid Sampling in Imbalanced Learning

Abstract

: Imbalanced data remain a critical challenge in classification, as skewed distributions bias models toward majority classes and diminish sensitivity to minority classes, which are often the most critical. To address this issue, this paper proposes the Information Filtered Hybrid Algorithm (IF-HA), a novel entropy-based sampling method that integrates undersampling and oversampling guided by information theory. IF-HA quantifies instance importance through an instance-wise difference statistic. In the undersampling stage, majority of instances with low difference statistics in the border area are eliminated, while in the oversampling stage, synthetic samples are generated from two minority core points or two minority instances with high difference statistics located in the border area. This process removes noise, eliminates redundant majority border points, and generates synthetic minority samples in informative regions until an entropy-based imbalance threshold is reached. The proposed algorithm is evaluated on 20 benchmark datasets from the UCI and KEEL repositories. Results demonstrate that IF-HA consistently improves minority detection and achieves higher F1 Scores, recall, and AUC (Area Under the Curve) than other methods, including SMOTE, Borderline-SMOTE, ADASYN (Adaptive Synthetic Sampling), and SMOTE-TLNN-DEPSO. A real-world tuberculosis (TB) dataset from Indonesia was further used to validate the practical applicability of IF-HA using KNN, Random Forest, and XGBoost (eXtreme Gradient Boosting) classifiers. The results show consistent improvements after applying IF-HA. These findings indicate that entropy-based hybrid sampling is a promising approach for structured tabular imbalanced classification, while further validation on high-dimensional text and image datasets remains necessary to establish broader generalizability.