Variable Selection under Multicollinearity and Heavy-Tailed Errors: A Systematic Assessment of Classical and Penalized Methods
Abstract
Variable selection in high-dimensional regression becomes particularly challenging when multicollinearity and heavy-tailed errors occur simultaneously. This study systematically evaluates the performance boundaries of classical, shrinkage, sparse, and robust regression methods under these conditions. A Monte Carlo simulation experiment was conducted using 100 candidate predictors, of which 10 were truly active, while predictor correlation was varied across ρ = 0.1, 0.3, 0.6, and 0.9. Three error environments normal, contaminated normal, and Student-tt heavy-tailed errors were considered. Ordinary Least Squares (OLS), Ridge, LASSO, Elastic Net, and Huber regression were evaluated using mean squared error (MSE), mean absolute error (MAE), root mean squared error (RMSE), number of selected variables, true positive rate (TPR), false positive rate (FPR), coefficient variability, and selection instability. The findings reveal a clear trade-off between predictive accuracy, sparsity, robustness, and selection stability. OLS and Ridge achieved comparatively favorable predictive performance in several scenarios but retained all 100 predictors, resulting in substantial over-selection. LASSO achieved greater sparsity and relatively lower false-positive rates but became increasingly unstable under severe multicollinearity, with selection instability rising to 0.2976 at ρ = 0.9. Elastic Net demonstrated the most balanced performance, maintaining high signal recovery while substantially reducing selection instability as predictor correlation increased. Under contaminated errors, Elastic Net also outperformed LASSO in signal recovery and prediction. Huber regression reduced sensitivity to extreme observations but did not provide sparse variable selection. Overall, the results demonstrate that no conventional method simultaneously addresses heavy-tailed errors, multicollinearity, and sparse selection adequately. Elastic Net provides the strongest benchmark among the evaluated procedures, while the findings motivate the development of a robust, sparsity-inducing, and correlation-aware variable-selection framework for high-dimensional regression.