Comparative Performance of Modern Regression Techniques for Predictive Modelling of High-Dimensional Clinical Data in e-Health Systems

Abstract

This paper examined the relative performance of traditional and modern regression methods when used in high-dimensional data where there are many features and the predictive signal is weak. Their main aim was to do a rational comparison of the linear, regularised, ensemble, and deep learning regression models on a single and reproducible experimental setup. An experimental, high-dimensional, real-world dataset of about 200,000 observations and 200 numerical covariates was used to assess the behaviour of the models in realistic conditions. The use of multiple regression methods was applied, and they are ordinary least squares, ridge, lasso, elastic net, random forest, gradient boosting, XGBoost, and a neural network-based regressor. Error-based measures like mean squared error, root mean squared error (RMSE), mean absolute error (MAE), and goodness-of-fit measures were used as a measure of model performance and computed metrics, training and inference time. The cross-validation was selectively used on the computationally efficient models, and models that were more complex were tested by a hold-out strategy. These findings demonstrated that regularised linear models (especially lasso and elastic net) were always able to perform moderate predictive performance with high stability and low computational cost. Conversely, ensemble and deep learning models failed to provide significant accuracy improvement, although they required much more computation. The results emphasise that better performance of high-dimensional regression with more models is not always well-correlated with their complexity. Generally, the work can be viewed as offering applications of selecting a regression model in high-dimensional data with a focus on regularisation, robustness, and efficiency and providing researchers and practitioners with guidelines.