A Statistical Analysis of Classical and Advanced Regression Models in Data Science
TL;DR
It is concluded that advanced models can outperform classical baselines when the underlying data structure is complex, while classical models remain essential for interpretation, comparison, and model selection.
Abstract
Regression models remain foundational tools of both mathematical statistics and modern applied data science for prediction, explanation, and model comparison. In applied statistical modelling there is always a fundamental practical trade-off that must be considered. Classical regression models are attractive mainly because they are easy to interpret, and are built on a well-established statistical foundation. However, they will often describe real world data very poorly when the underlying relationships are curved, interactive, or simply more complex than a simple straight-line structure. More flexible models can often deliver substantially improved prediction performance in these cases, but this improvement almost always comes at the cost of reduced interpretability and transparency. This study presents a systematic comparative analysis of classical, regularized, nonlinear, and ensemble models across two widely used benchmark datasets: the Advertising continuous sales prediction task and the Diabetes binary medical classification task. All models were evaluated using cross-validation and held-out test-set performance. Third-degree Polynomial Regression produced the strongest overall result on the Advertising dataset, with a test R² of 0.9907. Gradient Boosting achieved the highest performance on the Diabetes dataset with a test ROC-AUC of 0.8315, though Logistic Regression remained a highly competitive and far more interpretable baseline. We conclude that advanced models can outperform classical baselines when the underlying data structure is complex, while classical models remain essential for interpretation, comparison, and model selection.