A framework for fair and robust clinical risk prediction through collaborative learning and localized uncertainty quantification

Abstract

Clinical prediction models trained on heterogeneous populations may exhibit disparities in predictive performance across patient subgroups, potentially reflecting differences in data availability, feature completeness, and representation in training data. Existing fairness-aware approaches can introduce performance-fairness trade-offs and commonly optimize group-level disparities without explicitly accounting for their subgroup-specific sources, limiting both their effectiveness and their clinical interpretability. This dissertation makes two methodological contributions. The first is a collaborative learning framework that treats demographic subgroups as clients and aggregates their model parameters using strategies that encode different assumptions about group contribution. Rather than optimizing an explicit group-fairness constraint, the framework preserves and integrates subgroup-specific information to improve equity while largely maintaining subgroup predictive performance. The second contribution is the localized conformal classifier, a neighborhood-adaptive conformal prediction framework for instance-level uncertainty quantification. By calibrating prediction sets according to the density and composition of a patient's local neighborhood in the feature space, the method communicates both the model's prediction and the uncertainty associated with that individual prediction. This information is particularly consequential when predictions inform clinical decision-making. Both contributions are evaluated across three clinical prediction tasks: 30-day mortality in ICU patients with sepsis, treatment non-completion in patients with substance use disorder, and hospital readmission after surgical resection in patients with colorectal cancer. Compared with reweighting and constrained optimization, collaborative learning achieves more favorable fairness-performance trade-offs with less degradation in predictive performance, and its variants consistently appear on the Pareto frontier across all three prediction tasks. A novel difficulty decomposition framework, derived from the localized conformal classifier's calibration scores, distinguishes sources of uncertainty arising from the local neighborhood from those specific to an individual prediction. SHAP attribution reveals how collaborative learning and fairness interventions alter the feature-attribution patterns in the underlying prediction models. Surrogate regression trees trained on the localized conformal classifier's difficulty scores relate neighborhood and instance difficulty to clinical features observable at the point of care. Clinical features remain strongly associated with instance-level difficulty under collaborative learning, whereas these associations weaken substantially after reweighting and are absent in some analyses. Across all three clinical contexts, local outcome heterogeneity is the most consistent driver of neighborhood-level predictive difficulty, while proximity to the decision boundary is the principal driver of instance-level difficulty. Together, these contributions provide a framework for developing and evaluating clinical prediction models that are not only accurate, but also more equitable across patient groups and more informative about the reliability of individual predictions.