Jul 2026· Innovation Science and Technology· Vol 5, pp. 26-44· 0 citations· 32 references
TL;DR
A thorough investigation of ten representative classical machine learning algorithms is presented, constructing a complete and logically coherent knowledge system of classical machine learning, serving as an accessible and pragmatic reference for novice learners and engineering practitioners.
Abstract
Classical machine learning algorithms constitute the fundamental cornerstone of modern data science and intelligent system development. While deep learning has achieved transformative breakthroughs across numerous fields in recent years, classical methods remain indispensable in practical scenarios characterized by limited training data, stringent interpretability requirements, or constrained computational resources. Nevertheless, existing studies generally lack a systematic, unified, and beginner-friendly comprehensive survey that integrates theoretical elaboration, multi-dimensional comparative analysis, and actionable algorithm selection guidance.
To fill this research gap, this paper presents a thorough investigation of ten representative classical machine learning algorithms: Linear Regression, Logistic Regression, Decision Tree, Random Forest, Support Vector Machine, K-Nearest Neighbors, Naive Bayes, Adaptive Boosting, K-Means, and Principal Component Analysis. Each algorithm is explicated within a consistent structural framework, covering its mathematical formulation, core working mechanism, distinct advantages, inherent limitations, and typical application scenarios. Furthermore, a horizontal comparative analysis is conducted across seven critical dimensions, and a practical algorithm selection framework with targeted recommendations for typical industrial scenarios is proposed.
This work constructs a complete and logically coherent knowledge system of classical machine learning, serving as an accessible and pragmatic reference for novice learners and engineering practitioners. It also provides insights into future integration trends between classical algorithms and emerging technologies including large language models, explainable artificial intelligence, edge intelligence, automated machine learning, and federated learning.
High-dimensional data are now ubiquitous in the modern science and industry, such as bioinformatics, text mining, computer vision, finance, and cybersecurity. A prominent feature of such data is having many features in comparison with the number of observations, which may cause the judgement problem of the curse of dimensionality, greater computational cost, feature overlap, and overfitting. Though supervised learning algorithms are extensively used to do predictive modeling, they have very different performance properties in high dimensional feature space. The paper contains a thorough comparison of some of the most popular supervised learning algorithms in the high-dimensional data analysis scenario. The paper provides a systematic comparison between the linear, non-linear, probabilistic, and ensemble-based classifiers, which are: Logistic Regression, Support Vector Machine, k -Nearest Neighbor, Decision Tree, Random Forest, Naive Bayes, and Artificial Neural Network. Special attention is given to the study of the behaviour of an algorithm based on scalability, ability to generalize, resistance to noise, feature sparsity, and interpretability. Besides, the paper explores how dimensionality reduction and feature selection methods impact on the performance of classification. It suggests a single experimental procedure with standardized preprocessing pipelines, cross-validation schemes and performance metrics accuracy, precision, recall, F1-score and cost of the computation. To give the concept theoretical background, mathematical formulations of learning objectives and decision functions are given. The comparative analysis indicates that there is no universal algorithm that has the best performance in all high-dimensional conditions; the performance highly depends on the sample size, the features correlation, the level of data distribution as well as noise. This study has practical implications on researchers and practitioners to consider the proper supervised learning model to use the high-dimensional datasets and identifies future research opportunities in scalable and interpretable learning.
S. Verma, Naveen Kumar· International Journal of Mac...· 0 citations
Machine Learning (ML) has deeply reinvented artificial intelligence, translating from an academic norm to becoming the basic engine of modern AI. Its milestone or achievement is paramount in various fields of endeavor as healthcare, autonomous systems, finance, and scientific discovery etc. This paper examined a comprehensive, systematic literature review of the three main machine learning paradigms: Supervised Learning (SL), Unsupervised Learning (UL), and Reinforcement Learning (RL). Following a PRISMA guiding principle and harmonizing peer-reviewed studies published basically between 2018 and 2026, this review carefully states the theoretical groundwork of algorithmic evolution, learning mechanisms, evaluation methodologies, and practical applications of each paradigm. This review establishes a comparative analytical framework that shed light on complex inherent trade-offs in data dependency, computational complexity, and applicability. Supervised Learning is explored in the perspective of its foretelling dominance and the incipient shift toward self-supervised and semi-supervised techniques to mitigate label scarcity. Unsupervised Learning is explored for its utility in pattern discovery and dimensionality reduction across increasingly large, unlabeled datasets. Reinforcement Learning is re-examined as a maturing field for sequential decision-making, worth over $122 billion in 2025, and integrated with deep learning for complex control tasks. A central focus is placed on the growing convergence of paradigms through hybrid approaches such as self-supervised, semi-supervised, and deep reinforcement learning. The study integrates cross-paradigm challenges related to data efficiency, model interpretability, scalability, robust evaluation, and ethical deployment, while appraising emerging frontiers such as meta-learning, federated learning, continual learning, and multimodal learning. The review culminates in pinpointing actionable research gaps and given a structured decision framework to direct researchers and experts in choosing and synergistically merging ML paradigms for complex, real-world problems.
Francis Uwadia, M. I. Akazue, Efeobor Abel Edje· International journal of re...· 0 citations
Research that combines Evolutionary Machine Learning and Self-Supervised Learning has been steadily increasing in recent years, suggesting that the combination of these two areas can help both in shaping evolutionary processes and in automating the design of deep neural networks, while also reducing the need for labelled data. Yet, no survey details how these two areas are used together. To help with this, we introduce Evolutionary Self-Supervised Learning as a research area and propose a taxonomy based on two core directions, evolution applied to Self-Supervised learning and Self-Supervised learning applied to evolution. For each direction, we categorise and discuss different approaches, compare the performance of different works, and discuss observed trends and their impact on performance and computational cost. Following this, we identify six open challenges around pretext task design, ablation studies with label scarcity, and the design of reliable and low-cost fitness metrics that take into account self-supervised mechanics. Based on these challenges, we propose research directions for researchers within the field.
Adriano Vinhas, João Correia, Penousal Machado· IEEE Access· 2 citations
Predictive analytics plays a crucial role in data science by forecasting future trends using historical data. Traditional statistical techniques such as linear regression, logistic regression, ARIMA, and Bayesian inference provide interpretable and mathematically rigorous models but often struggle with large-scale, complex, and nonlinear datasets. Machine learning approaches, including Decision Trees, Support Vector Machines, Random Forests, Artificial Neural Networks, and Ensemble Learning, offer superior predictive capabilities but may lack interpretability and uncertainty estimation. To address these limitations, hybrid statistical–machine learning methods have emerged, combining statistical feature engineering, probabilistic modeling, and machine learning algorithms to improve prediction accuracy and robustness. This paper reviews the theoretical foundations, architectures, and applications of hybrid predictive models across finance, healthcare, manufacturing, transportation, and business intelligence. A generalized hybrid framework incorporating feature selection, model training, ensemble optimization, and validation is presented. The analysis indicates that hybrid approaches consistently outperform standalone statistical and machine learning models in terms of accuracy, reliability, and generalization. Key implementation challenges, including computational complexity, interpretability, data quality, and parameter optimization, are also discussed. The study concludes that hybrid statistical–machine learning models represent a promising direction for next-generation predictive analytics and intelligent decision-support systems.
Arvind Kumar Singh· International Journal of Mac...· 0 citations
Extreme learning machine (ELM) is a fast and generalizable learning model widely used in classification and regression tasks. Meta-ELM improves the performance of the ELM by combining multiple base ELMs, each trained on different data partitions. These base models were then used as hidden nodes for the parent ELM, forming a hierarchical architecture. Constrained ELM (CELM) improves upon standard ELM by replacing the random input weight assignment with weights derived from combinations of selected data samples. There are five known variants of CELM, each based on different combination strategies. This study proposes a novel model, the CELM, which integrates the Meta-ELM framework with the structured weight assignment of the CELM. Rather than using random weights, it constructs input weights from representative data combinations, thereby improving the learning stability and performance. The model was evaluated on 10 benchmark classification datasets, including adult census income, breast cancer, diabetes, dry bean, fashion MNIST, heart disease, raisin, rice, wine, and Modified National Institute of Standards and Technology. The results showed consistent improvements in accuracy, precision, recall, and F1 score. The Meta-difference extreme learning machine (DELM) variant of the proposed model outperforms standard ELM and Meta-ELM, while showing competitive results compared with various state-of-the-art models such as support vector machines, random forest, XGBoost, LightGBM, and CatBoost. These findings confirm the robustness of the proposed Meta-DELM, achieving an accuracy of 87.29% across all datasets.
Zafer Serin, C. Karakuzu, Ugur Yuzgec· Computer/law journal· 0 citations
Recent advances in machine learning have produced numerous predictive algorithms for classification, regression, and forecasting tasks. However, selecting the most suitable model for a specific dataset remains challenging, often requiring expert knowledge, extensive experimentation, and significant computational resources. To address this issue, automated model selection has emerged as an important research area within machine learning and intelligent decision-support systems. Meta-learning, or “learning to learn,” provides an effective solution by utilizing knowledge gained from previously analyzed datasets to predict the performance of learning algorithms on new datasets. It examines dataset characteristics, known as meta-features, and recommends appropriate machine learning models, thereby improving selection accuracy while reducing computational costs. This study proposes a comprehensive meta-learning framework for automated predictive model selection. The framework includes dataset characterization, meta-feature extraction, meta-dataset generation, algorithm evaluation, and meta-model construction. Statistical, information-theoretic, landmarking, and complexity-based features are used to describe datasets and train a meta-learning model capable of recommending suitable algorithms for new predictive tasks. The research evaluates several popular machine learning algorithms, including Decision Trees, Support Vector Machines, Random Forests, Naïve Bayes, Artificial Neural Networks, and k-Nearest Neighbor classifiers. Experimental results demonstrate that meta-learning significantly improves model recommendation accuracy compared to traditional trial-and-error approaches while reducing training time and computational overhead. As part of the broader field of Automated Machine Learning (AutoML), the proposed framework offers an intelligent algorithm recommendation system that supports efficient resource utilization and assists practitioners in selecting high-performing models without extensive machine learning expertise. The findings highlight the potential of meta-learning-based model selection for future intelligent analytics, decision-support, and large-scale data mining systems.
Pooja Agarwal, Rakesh Chandra· International Journal of Mac...· 0 citations