IQC: A Novel Criterion for Assessing Feature Selection Stability in High-Dimensional Analyses

Abstract

In recent years, the issue of reproducibility in scientific studies has been at the forefront of scientific discussion. Many of these issues arise due to poor employment of statistical methods, making it crucial to address these knowledge gaps for frequently used methods. Elastic net regression (ENET) is one such algorithm used frequently in biology and public health due to its ability to select important predictors (also known as “features”) from large data sets. However, ENET struggles with consistent feature selection and hyperparameter choice in these applications due to low observation counts and noisy data. This limitation hinders ENET’s effectiveness in biological research due to low model reproducibility and predictor interpretability. We propose the Instability Quotient Criterion (IQC) as a metric to quantify the consistency of selected features for feature selection models such as ENET, with a specific emphasis on biological applications. IQC leverages ensemble modeling and Shannon’s entropy in order to determine the stability of selected features derived from ENET models. We demonstrate IQC’s robustness via simulation studies, showing that IQC is capable of detecting significant decreases in model stability with lower observation counts, higher dimensionality, and higher proportions of noisy signal. We additionally utilize simulation to provide a naive threshold for low versus high stability. Finally, we applied IQC to a case study to illustrate how our method can be integrated into a larger analysis pipeline when looking at microarray gene expression data. IQC indicated stability issues for differentially-expressed genes when classifying acute leukemia subtypes in spite of high classification success. Lastly, we discuss how model instability impedes interpretation of biologically significant genes for discerning cancer subtypes. Overall, we present IQC as a robust evaluation metric that can easily be incorporated into high-dimensional biological analyses. Author summary Addressing the reproducibility crisis is paramount for bolstering confidence in scientific studies. In biological settings specifically, making consistent predictions when using machine learning techniques is vital. Methods such as elastic net regression (ENET) often suffer from inconsistent predictor selection when applied to biological data due to high dimensionality, noise, and low observation counts. Thus, the ability to detect when predictor selection is unstable is valuable for researchers in biology and health. We present the Instability Quotient Criterion (IQC) as a novel criterion to assess model stability for elastic net regression. This criterion gives a value between 0 and 1 that displays how consistently the same predictors are chosen. We find through simulation studies that IQC is robust in its detection of a stable versus unstable model when considering its selection of important predictors. We also demonstrate IQC’s usefulness in a larger modeling pipeline when applied to gene expression data. Our findings show that IQC is a valuable tool for inferring the robustness of biological predictors with more nuance and precision.