SoftMCC is a calibration-sensitive MCC-family selector with bounded stability and utility evidence, coupling an MCC-specific calibrated identity with a tie-aware, shared-pool selection protocol.
Abstract
Model selection for imbalanced binary classification often uses the Matthews correlation coefficient (MCC), but thresholding makes validation rankings threshold-dependent. SoftMCC is a post-training MCC validation framework on established probability-valued confusion counts, coupling an MCC-specific calibrated identity with a tie-aware, shared-pool selection protocol. Its core score is a covariance-normalized probability-label association, reduces exactly to MCC for hard predictions, and is Pearson-bounded. Under perfect population calibration it equals the Brier skill score with identical candidate ordering; outside that regime the gap does not identify calibration error. Across 18 settings with 12 duplicate-safe grouped repeats, SoftMCC attains the best stability mean rank (2.31) and highest mean tie-corrected Kendall's W (0.659), with a significant Friedman test (p=0.007); Nemenyi analysis separates it from AUPRC and MCC@0.5, while 14-source-family sensitivity retains only the latter. Selected-model utility shows no advantage. Three of six prespecified comparisons have negative mean test-MCC differences, only F1@best survives Holm correction (p=0.014), and the dataset-level test is not significant (p=0.117). Label permutation lowers mean W to 0.092; temperature scaling shifts SoftMCC rankings (mean Spearman 0.851) whereas rank-based and threshold-optimized metrics remain invariant. SoftMCC is a calibration-sensitive MCC-family selector with bounded stability and utility evidence.
Researchers using latent profile and latent class analysis (LPA/LCA) commonly assign individuals to their modal class without evaluating whether this simplification distorts reported class sizes, profile means, or high-severity subgroups. Existing classification-quality diagnostics-entropy, average posterior probabilities, and Masyn's odds of correct classification and classification probability-assess how sharply a model separates its classes, not whether hard-assigned summaries diverge from probability-weighted ones. We address this through an empirical benchmark (four-class SCL-90 solution; N = 59,408), a fully crossed simulation (class separation, class balance, and indicator-class discrimination precision; 27 conditions), and a six-index diagnostic framework. Hard-assigned and probability-weighted summaries were interchangeable under favorable conditions but diverged under low class separation, severe imbalance, or low precision-most acutely for the smallest, most extreme class. The framework offers simulation-calibrated thresholds for documenting assignment adequacy in continuous-indicator LPA; extension to categorical LCA requires further validation. Hard assignment is most defensible when its adequacy is documented rather than assumed.
Xiaohui Chen, Siguang Chen, Chenglin Wang et al.· Educational and Psychologica...· 0 citations
Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.
For hierarchical models, Pareto-smoothed importance-sampling leave-one-out cross-validation (PSIS-LOO) fails on the folds where a random-effect coordinate is data-driven and its group is small. We show that the Gelman-Pardoe pooling factor and structural leverage predict these folds from model structure and group sizes, without forming importance weights. In Gaussian linear mixed models the leverage reduces to group size, giving a design-time map that separates the failing ($\hat{k}>0.7$) folds with AUC 0.96; across replicated logistic GLMMs the post-fit, weight-free predictor reaches AUC 0.81. The cure is integrated importance sampling: marginalise the random-effect block and importance-sample only the base parameters. This is not new, but we contribute its observation-level specialisation for random-intercept GLMMs: an analytic Gaussian downdate and a 1-D quadrature for Bernoulli, binomial and Poisson responses, packaged as a drop-in rb_loo(fit). Against exact refits, this marginalised estimator (RB-LOO) is $3\times$ more accurate than moment matching on singleton-heavy logistic GLMMs. On overdispersed count data with 97 failing folds, moment matching leaves 37 uncorrected and is no more accurate than raw PSIS-LOO, while RB-LOO reproduces the 82-minute exact refit (elpd RMSE 0.04) at no cost. The error changes decisions: against a negative-binomial model, PSIS-LOO reports decisive evidence ($z=4.9$) and reloo reports significant evidence ($z=3.4$) for the more complex model, where an exact analysis, reproduced by RB-LOO, finds the two indistinguishable ($z=1.0$). A base-fiber Schur decomposition splits case-deletion influence into a vertical (pooling) term that governs where PSIS-LOO fails and a horizontal (variance-component) term that governs where RB-LOO is itself strained, giving a two-level triage that recovers the exact answer while refitting only the few folds that need it.
OBJECTIVE
Selective classification improves reliability by allowing models to abstain on uncertain inputs, which is critical in safety-sensitive domains such as healthcare. However, commonly used evaluation metrics obscure fairness issues under class imbalance, leading to disproportionately high rejection of minority classes and misleadingly favorable assessments of selective strategies.
METHODS
We introduce two imbalance-aware evaluation metrics, Class-Averaged AURC (CA-AURC) and Class-Averaged AUGRC (CA-AUGRC), which integrate risk against class-specific coverage and then average across classes, alongside the Area under the IAM-coverage curve (AUIC) as a complementary imbalance-aware performance metric. In addition, we propose a class-conditional coverage-matching selection strategy that enforces balanced rejection across diagnostic categories. The proposed framework is evaluated on a clinical complete blood count dataset comprising 3316 patient records, 11 laboratory features, and 9 diagnostic classes, and validated on three additional publicly available benchmark datasets covering different domains, class structures, and imbalance levels.
RESULTS
While conventional metrics such as AURC and AUGRC favor classical selective strategies, the proposed class-averaged metrics reveal substantial disparities in rejection behavior under class imbalance. Using CA-AUGRC and AUIC, the class-conditional strategy consistently outperforms both the classical approach and LABEL, a theoretically principled set-valued baseline, across all four datasets and five uncertainty measures. Rank-based statistical comparisons confirm significant advantages of the class-conditional strategy on CA-AUGRC (p=0.009, r=0.638) and AUIC (p<0.001, r=0.957).
CONCLUSION
The results demonstrate that both evaluation and selection in selective classification must be class-aware to ensure fairness and clinical usefulness. Class-averaged metrics and class-conditional selection provide a more reliable basis for assessing selective classifiers in imbalanced medical data, with consistent generalizability across diverse datasets and uncertainty measures.
Fatih Sağlam, Ünsal Özgen, Alper Uygun et al.· Journal of Biomedical Inform...· 0 citations
Given a patient's clinical findings, a diagnostic system ranks possible diseases and must decide when to endorse its first prediction or defer it for review. This decision is usually made by thresholding the top score. Selective prediction over ranked outputs begins with two checks. First, the ranker must produce enough correct top-ranked predictions to make the target feasible. Across 2,000 patient records stratified by disease prevalence, eight small open-weight LLMs achieve at most 4.6% Recall@1 on ultra-rare diseases. At 10% coverage, even a perfect confidence ranking of their existing predictions therefore cannot reach 50% selective accuracy. More accurate models pass the same check, showing that the limit is regime-specific. Second, the confidence signal must match the decision being made. For fixed-candidate rankers, the top-two margin cancels components shared across candidates. On phenotype-only Exomiser, it selects 10% of cases at 29.0% accuracy, compared with 13.3% overall, while the top score provides no reliable gate. Yet that cancellation can remove information needed to detect whether the candidate list contains an answer. SciFact retrieval and biomedical entity linking confirm this distinction. Finally, we prove that unlabelled scores alone cannot determine whether switching to the margin will help.
Zhaoyang Jiang, Zhizhong Fu, Yunsoo Kim et al.· 0 citations
Long-tailed classification poses a reliability challenge because models trained on imbalanced data are unevenly reliable across frequent and underrepresented classes. While existing methods address imbalance through re-balancing, adjustment, representation learning, or multi-expert modeling, they rarely estimate which expert should be trusted for each class. This paper proposes CLEAR (Class-wise reLiability-aware Expert Aggregation for long-tailed Recognition), a modular ensemble framework for long-tailed classification. CLEAR generates diverse experts through threshold-based structured sampling while preserving the full label space, then estimates a class-wise trust score for each expert using a smoothed class-wise precision formulation. During inference, expert predictions are combined through class-wise generalized product-of-experts aggregation, allowing different experts to be emphasized for different classes. Experiments on CIFAR-100-LT, ImageNet-LT, and Places-LT across multiple backbones show that CLEAR achieves competitive overall accuracy and particularly strong few-shot performance. These results support class-wise expert reliability as a useful design principle for long-tailed ensemble learning.