Beyond Predictive Accuracy: A Multi-Criteria Adaptive Framework for Reliability-Aware Machine Learning Model Selection in Cardiovascular Disease Prediction

Abstract

Machine learning (ML) models are increasingly being used for cardiovascular disease prediction with model selection often guided by predictive performance measures such as accuracy. However, predictive accuracy per se may not adequately reflect the prediction reliability relevant to clinical decision support, especially when competing models differ in calibration, predictive uncertainty, confidence and decision reliability. The study proposes MASE-RAMR, a Multi-Criteria Adaptive Selection Engine with Reliability-Aware Model Ranking, to offer a wider basis for evaluation and ranking of cardiovascular disease prediction models. The proposed framework clusters heterogeneous evaluation metrics into four reliability-oriented meaningful dimensions: Accuracy, Calibration, Uncertainty, and Decision Reliability. Then, a hierarchical CRITIC-based weighting strategy is adopted, first the importance of the specific metrics is determined in each dimension and then the global importance of the resulting dimensions is estimated. The obtained adaptive criteria matrix is then evaluated by TOPSIS to compute Reliability Aware Model Ranking (RAMR) score for each candidate model. For the evaluation of the framework, six cardiovascular disease datasets were used and compared with the conventional CRITIC-TOPSIS model ranking approach with the same experimental setting. The comparison demonstrates two important behaviours of the proposed framework, decision stability, in which the same top-ranked model is retained for a number of datasets, and adaptive ranking, in which the proposed framework changes or refines the model rankings when the additional reliability dimensions provide discriminative information. Further analysis of traditional classifiers shows that the framework does not inherently favour complex ensemble or boosting models, as conventional classifiers achieve the highest reliability rankings on several datasets. These results show that MASE-RAMR provides a dataset-dependent and reliability-aware strategy for ML model selection beyond predictive accuracy alone, providing a more comprehensive basis for identifying candidate models for clinical decision support.