Novel Regularization Methods to Prevent Overfitting in Machine Learning Models

Abstract

The problem of overfitting is one of the most persistent in the current machine learning (ML), especially as models and data dimensionality increases. Although classical regularization methods like L1, L2, dropout, and early stopping have been proven to be efficient, their weakness is realized in large-scale, deep, and data-sparse learning settings. In the current paper, a detailed study has been carried out on new regularization techniques that aim to enhance the performance of generalization and, at the same time, ensure the expressiveness of the model. We present a single taxonomy of the new regularization methods such as adaptive regularization, information-theoretic constraints, structured sparsity, stochastic regularization and regularization at the representation level. Moreover, we suggest Hybrid Adaptive Information Regularization (HAIR) which is a dynamic complex/generalization balance which is regularized by entropy-based penalties and parameter-sensitivity analysis. Numerous comparative studies show that the suggested approach is more effective than the traditional approaches in various learning paradigms. The findings have emphasized the importance of advanced regularization in developing robust, scalable and interpretable ML systems. The current study provides a certain contribution to both theoretical background and methodological developments as well as empirical findings in favor of next-generation regularization approaches.