Nov 2026· Engineering and Technology Journal· 0 citations
TL;DR
Tree-based models exhibited the largest reductions in accuracy and the steepest increases in calibration error under scarcity, whereas logistic regression preserved threshold balance and probability calibration at negligible computational cost.
Abstract
Machine learning research is frequently conducted under an implicit assumption of data abundance, yet applied software development often proceeds under severe data constraints. This study evaluated how five widely used classification algorithms—logistic regression, support vector machines, Gaussian naive Bayes, decision trees, and random forests—degrade when training data are systematically reduced. Eight tabular datasets drawn from the UCI Machine Learning Repository, spanning small, medium, and large volume tiers, were analysed within a quantitative comparative experimental design. Training folds were reduced to 100%, 75%, 50%, and 25% of their original size through stratified fractional sampling inside a stratified five-fold cross-validation loop, while validation folds were retained at full volume. Ten criteria were recorded: accuracy, precision, recall, F1 score, AUC-ROC, AUC-PR, mean absolute error and mean squared error of predicted probabilities, training time, and interpretability. One-way analyses of variance and Tukey honestly significant difference tests were applied at an alpha level of .05. Between-model differences were statistically significant in 31 of 32 dataset-by-volume configurations. Tree-based models exhibited the largest reductions in accuracy and the steepest increases in calibration error under scarcity, whereas logistic regression preserved threshold balance and probability calibration at negligible computational cost. Random forests attained the highest scores only when data were abundant. Pairwise comparisons aggregated across datasets were not significant, indicating that dataset context, rather than architectural complexity alone, governs absolute performance.
The processing of big data has become a top priority, especially with the emergence of machine learning algorithms. Standard models suffer from several limitations and lack robustness. This paper studies four machine learning models: SVM, XGBoost, Random Forest, and logistic regression, applied to the Titanic dataset for binary classification. Model performance is evaluated using the following metrics: precision, recall, accuracy, F1-score, and area under the ROC curve (AUC), as well as through confusion matrix analysis. We applied a 5-fold cross-validation method, and statistical significance was assessed using the Wilcoxon test. Based on the analysis of the results, logistic regression achieved the best performance on the test set in terms of precision and AUC of the ROC curve, while Random Forest and XGBoost provided competitive and stable results across the validation folds. SVM also performed well, but it was slightly less effective. The Wilcoxon test shows no statistically significant difference between the models (p > 0.05). Examination of the confusion matrix reveals a tendency for ensemble methods to minimize false negatives. Overall, logistic regression demonstrates strong and good generalization ability, while ensemble methods constitute robust alternatives for binary classification on structured data.
Zahra Benider, H. Bouzahir, Jaafar Idrais· EPJ Web of Conferences· 0 citations
We compare a radial-basis-function support vector machine (SVM) with a Gaussian naive Bayes (NB) classifier on a reproducible synthetic binary task designed for controlled evaluation. The data matrix contains 1,000 samples and 20 features, of which 15 are informative and 5 are redundant linear combinations; class labels include a controlled flip rate of 0.1. Models were fitted in scikit-learn 1.3.0 (SVM: RBF kernel, C = 1.0, gamma = 'scale'; NB: GaussianNB with default variance smoothing) under an 80/20 stratified hold-out and 5-fold stratified cross-validation. On the test partition, NB attained accuracy 89.95%, precision 91.35%, and F1-score 90.48%, while SVM attained 89.45%, 90.48%, and 90.05%, respectively; recall was identical at 89.62%. McNemar's test (p = 0.8231) and Cohen's kappa (NB 0.7984; SVM 0.7882; Δκ = 0.0102) indicate that the two predictors are statistically equivalent in discriminative accuracy. The practical distinction is computational: NB trained 22.5× faster, predicted 12× faster, used 2.9× less memory, and produced a 12× smaller model file. Under resource limits, therefore, NB is the preferred choice even though predictive accuracy does not differ significantly from SVM.
P. Shanthini, S. Parthasarathy· New Mathematics and Natural...· 0 citations
Skewed class distribution continues to be one of the central obstacles in binary classification, since a learning model tends to lean toward the dominant class and consequently overlooks observations belonging to the under-represented class. The purpose of this research is to examine how the choice of distance measure inside SMOTE, specifically Euclidean, Manhattan, Chebyshev, and Hamming, affects predictive quality on imbalanced binary data. Ten publicly available binary datasets drawn from the KEEL repository, whose imbalance ratios span from 1.86 up to 15.80, were used in the experiment. Every dataset was preprocessed and partitioned into 80% for training and 20% for testing; oversampling with SMOTE was carried out on the training portion only, after which four learners, namely Naive Bayes, Decision Tree, Logistic Regression, and k-Nearest Neighbor, were assessed. Model quality was judged through the Matthews Correlation Coefficient (MCC) together with the G-Mean, as these two indicators describe imbalanced performance more faithfully than plain accuracy. The comparison revealed that pairing Euclidean-based SMOTE with Logistic Regression yielded the strongest average scores (MCC = 0.72; G-Mean = 0.79); Manhattan-based SMOTE reached its top MCC again with Logistic Regression (MCC = 0.68) and its top G-Mean with the Decision Tree (G-Mean = 0.79); Chebyshev-based SMOTE delivered the best overall combination together with the Decision Tree (MCC = 0.74; G-Mean = 0.84); and Hamming-based SMOTE performed best alongside Logistic Regression (MCC = 0.73; G-Mean = 0.81). Taken together, these outcomes suggest that the distance function chosen within SMOTE shapes the quality of the generated synthetic points and, in turn, the behavior of the trained classifier.
F. Y. Pamuji, Luthfi Indana, Mohammad Dwi Irfan Affandi· Computer Science (CO-SCIENCE...· 0 citations
The growing need for accurate, consistent, and reliable loan approval systems The use of machine learning in credit decision-making is increasingly important for financial institutions, but comparative research still often focuses on Accuracy or a limited number of classification metrics, so the trade-off between predictive performance and computational efficiency is not fully described. This study aims to develop a Standardized Multi-Metric Evaluation Framework (MMEF) to support the selection of more objective and reproducible machine learning models in the case of loan approval. The research method uses a standardized experimental pipeline with consistent preprocessing, class balancing using the Synthetic Minority Over-sampling Technique (SMOTE), identical data sharing, model optimization, and multi-metric evaluation. Five algorithms, namely Logistic Regression, Support Vector Machine (SVM), Random Forest, XGBoost, and CatBoost, are compared using a loan approval dataset consisting of 45,000 records and 13 predictor features. The evaluation includes Accuracy, Precision, Recall, F1-score, ROC-AUC, training time, and Overall Score. The results show that XGBoost provides the best overall performance with Accuracy 87.86%, Precision 66.77%, Recall 90.30%, F1-score 76.77%, ROC-AUC 96.27%, and Overall Score 0.827503. CatBoost has the fastest training time of 1.01 seconds, while SVM obtained the highest Recall of 92.40% with a much longer training time. These results indicate that model selection is not sufficient based on a single metric. MMEF provides a more systematic evaluation basis to identify models that have a balance of performance and efficiency in loan approval experiments.
Trihartono Agus, Agus Ilyas Ilyas, S. Sattriedi et al.· Jurnal Informatika: Jurnal P...· 0 citations
Hybrid machine learning offers a flexible approach for predictive analytics and intelligent decision-making in heterogeneous data environments. This study developed and evaluated Logistic Regression, Random Forest, soft voting, and weighted voting using a publicly available educational dataset comprising 4,424 student records, 36 predictors, and three outcome classes: dropout, enrolled, and graduate. Data preprocessing included categorical encoding, numerical scaling, class-weighted learning, and stratified training-testing separation. The models were compared using accuracy, balanced accuracy, macro precision, macro recall, macro F1-score, receiver operating characteristic area under the curve, log loss, and calibration measures. The weighted-voting ensemble achieved the best overall performance, with 76.72% accuracy, 71.89% balanced accuracy, 71.81% macro F1-score, and an area under the curve of 0.904. Performance improved from a macro F1-score of 55.54% with baseline predictors to 65.91% after first-semester information and 71.81% after second-semester information were added. Academic progression indicators, tuition-fee status, age at enrolment, debtor status, course, and gross domestic product were influential predictors. Probability thresholds supported high-priority intervention, moderate-risk monitoring, and uncertain-case review. The enrolled category remained the most difficult outcome to classify because of imbalance and transitional characteristics. Hybrid ensembles provided accurate, interpretable, and decision-oriented predictions, although external validation and prospective evaluation are required before institutional deployment.
Haleeful Jud, D. K. Gupta, Rupesh Kumar et al.· Journal of Intelligent Decis...· 0 citations
Abstract This study evaluates the performance of selected machine learning methods, Maximum Likelihood (MLC), Random Forest (RF), Extreme Gradient Boosting (XGB), Support Vector Machine (SVM), and Artificial Neural Networks (ANN), for land use/land cover (LULC) classification using Sentinel-2 satellite imagery. Each algorithm was tested across multiple classification scenarios that systematically varied both training sample size and training sample quality. To emulate realistic imperfections in reference data, controlled levels of label noise were introduced into the training set, and the resulting changes in classification performance were analysed. In addition, each model’s susceptibility to overfitting was assessed by comparing performance on training data and independent test data. To reduce the influence of a single random draw of training and testing pixels, all sampling-based experiments were repeated 10 times using different random seed values, and the reported results were summarized using repeated-run statistics. This design enabled a comparable assessment of how classification accuracy, stability, and generalization depend on dataset size and quality. The results indicate that SVM was the most consistent and reliable method across the tested scenarios, achieving high classification accuracy over a wide range of training sample sizes and maintaining strong robustness under noisy labels. Compared with the other methods, SVM showed lower sensitivity to degraded training data and a smaller tendency to overfit, which makes it a strong baseline choice when reference data are limited or imperfect.
P. Kupidura, Michał Szkibiel· Reports on Geodesy and Geoin...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 17, 2026
A USAF cadet and a Lincoln Laboratory researcher found AI chatbots can help nontechnical service members produce viable software applications for their unique problems.
