A Comparative Study of Supervised Learning Algorithms for High-Dimensional Data

Abstract

High-dimensional data are now ubiquitous in the modern science and industry, such as bioinformatics, text mining, computer vision, finance, and cybersecurity. A prominent feature of such data is having many features in comparison with the number of observations, which may cause the judgement problem of the curse of dimensionality, greater computational cost, feature overlap, and overfitting. Though supervised learning algorithms are extensively used to do predictive modeling, they have very different performance properties in high dimensional feature space. The paper contains a thorough comparison of some of the most popular supervised learning algorithms in the high-dimensional data analysis scenario. The paper provides a systematic comparison between the linear, non-linear, probabilistic, and ensemble-based classifiers, which are: Logistic Regression, Support Vector Machine, k -Nearest Neighbor, Decision Tree, Random Forest, Naive Bayes, and Artificial Neural Network. Special attention is given to the study of the behaviour of an algorithm based on scalability, ability to generalize, resistance to noise, feature sparsity, and interpretability. Besides, the paper explores how dimensionality reduction and feature selection methods impact on the performance of classification. It suggests a single experimental procedure with standardized preprocessing pipelines, cross-validation schemes and performance metrics accuracy, precision, recall, F1-score and cost of the computation. To give the concept theoretical background, mathematical formulations of learning objectives and decision functions are given. The comparative analysis indicates that there is no universal algorithm that has the best performance in all high-dimensional conditions; the performance highly depends on the sample size, the features correlation, the level of data distribution as well as noise. This study has practical implications on researchers and practitioners to consider the proper supervised learning model to use the high-dimensional datasets and identifies future research opportunities in scalable and interpretable learning.