It is found that this test to judge whether a computer program predicts whether a genetic variant causes disease is easier to pass than it looks, and about half the advantage held by programs trained on clinical data disappears once the gene pattern is removed.
Abstract
Missense pathogenicity predictors are routinely benchmarked against ClinVar, whose labels are strongly structured by gene: genes under diagnostic scrutiny accumulate pathogenic submissions while incidentally sequenced genes accumulate benign ones. We asked how much of a benchmark score this structure alone can produce. On 197,904 ClinVar missense variants validated against UniProt canonical sequences, a null model using no variant-level information, scoring each variant only by the pathogenic fraction of its own gene, reaches an area under the receiver operating characteristic curve (AUROC) of 0.921 under a random 10-fold split. On a common intersection of 169,989 variants, four current predictors exceed it by only 0.036 to 0.044. The inflation is not uniform, so it does not cancel when predictors are compared: under within-gene evaluation the ranking inverts, AlphaMissense rising from third to first and gMVP falling to third (p < 0.0001). The inversion survives removal of ceiling genes and replicates on an independently curated benchmark. Because both rankings derive from the same ClinVar labels, we arbitrated between them using data with no gene-level structure: agreement with 47 human deep mutational scanning assays matches the within-gene ranking and inverts the conventional one (p = 0.027, 0.0023). Across twenty-two dbNSFP predictors scored on one common intersection of 112,248 variants, with each tool’s exposure to clinical labels registered before any score was extracted, predictors never trained on such labels sit 0.051 AUROC behind supervised ones globally but only 0.026 behind within genes (difference +0.025 [+0.023, +0.027], p < 0.0001). Leave-one-out correction, the standard remedy, is worth 0.002 AUROC. Much of ClinVar benchmark performance reflects gene identity rather than variant effect, and the distortion changes which predictor a benchmark ranks first, in a direction experimental data contradicts. We release genenull, a single-file implementation, so reporting this baseline costs one function call. Author summary When a computer program predicts whether a genetic variant causes disease, we judge it by testing it against ClinVar, a public archive of variants clinicians have already interpreted. We found that this test is easier to pass than it looks. Some genes appear in ClinVar because they are suspected of causing disease, so most of their recorded variants are harmful; others are sequenced incidentally, so most of theirs are harmless. A program that knows nothing about a variant except which gene it sits in can exploit that pattern, and scores almost as well as the best tools available. This matters beyond a single number. When we removed the gene pattern and ranked variants inside a single gene, the order changed: the tool that looked best became worst, and the one that looked worst became best. Laboratory experiments that measure the effect of every possible variant in a protein agree with the new order, not the old one. Across twenty-two prediction tools, about half the advantage held by programs trained on clinical data disappears once the gene pattern is removed. We release software so anyone can measure this baseline in one line of code.
Missense pathogenicity predictors such as AlphaMissense are increasingly used in clinical variant interpretation, yet they are trained on germline labels dominated by loss-of-function (LOF) variants. Using an openly licensed, reproducible benchmark of 768 Cancer Gene Census genes scored with 49 predictors (labels from CIViC, COSMIC, cancerhotspots, ClinVar and gnomAD), we show that 42 of 49 tools (86%) score oncogene, gain-of-function (GOF) variants worse than tumour-suppressor variants. This under-scoring is mechanistically characterized: missed drivers occupy low-conservation, solvent-exposed, non-destabilizing positions (phyloP 2.51 versus 7.89; relative solvent accessibility 0.671 versus 0.185; gene-clustered p = 4.8×10⁻²⁰ and 2.3×10⁻³⁵), and, counter-intuitively, the unsupervised and protein-language models now entering clinical use are the most affected. Per-gene oncogenic thresholds span 0.07–0.99, so a single global cut-off is mis-calibrated for most genes; we provide a per-gene calibration map. A cancer-calibrated stack (OncoCal) modestly improves discrimination over the best single tool (AUROC ≈ 0.93 versus 0.87), rescues drivers such as JAK2 V617F (0.334→0.57), and generalizes to independent deep mutational scanning data. We provide an openly licensed framework to interpret and recalibrate these tools in the somatic setting rather than a replacement predictor.
Background: Variant-based pathogenicity predictors such as REVEL evaluate missense variants in isolation, discarding the gene-length and allele-frequency context needed to compare collections of genes. Methods: We introduce a composite gene-level metric integrating Hardy–Weinberg heterozygosity, coding-sequence length, and REVEL scores. The metric returns a single value per gene expressing variant burden per unit of coding sequence within a given cohort, so that the ratio between a case and a control cohort quantifies gene-level enrichment. It was evaluated on 55 high-confidence autism genes, defined as the intersection of three large-scale ASD sequencing studies, against the 1000 Genomes reference. Results: It identifies elevated pathogenic burden in 48 of 55 genes, removes gene-length and variant-count confounds, and substantially outperforms naive gene-level aggregation of REVEL scores. Bootstrap resampling and a label-permutation control confirm the enrichment is stable and not an artefact of the scoring construction. Conclusions: The metric allows genes to be ranked within a set and aggregate burden to be compared across gene sets. We present it as a complementary gene-level layer for case–control and gene-set comparisons, with a nonlinear successor outlined as future work.
PURPOSE
Clinical guidelines for interpreting genetic variants in the context of Mendelian disease require converting the outputs of pathogenicity prediction tools into well-calibrated probabilities. However, the existing calibration method is only valid when pre-committing to one tool, preventing clinical laboratories from using multiple tools with complementary strengths. To lift this restriction, we introduce Pathogenicity K-Nearest Neighbors (P-KNN), a flexible method that jointly calibrates any set of tools.
METHODS
P-KNN represents each variant in a multidimensional space defined by tool scores and estimates the probability of pathogenicity based on the proportion of pathogenic neighbors. We compared P-KNN against standard single-tool calibration of multiple predictors and meta-predictors at four historical time points.
RESULTS
P-KNN outperforms standard calibration of single tools and meta-predictors in two aspects: i) overall evidence strength and ii) alignment of the calibrated probabilities with true pathogenicity frequencies. Additionally, the evidence from P-KNN keeps improving with the addition of newer tools. It also correctly integrates correlated computational and experimental evidence that is overestimated by existing protocols.
CONCLUSION
P-KNN provides robust joint calibration for any set of pathogenicity prediction tools, thereby alleviating the constraint of pre-committing to a single predictor while enhancing statistical rigor and diagnostic yield. P-KNN is available via command line (https://github.com/Brandes-Lab/P-KNN) and precomputed scores (https://huggingface.co/datasets/brandeslab/P-KNN).
Po-Yu Lin, N. Brandes· Genetics in Medicine· 0 citations
Computational models of biology are ranked by averaging one prediction against many experimental realizations of a phenotype that are treated as interchangeable. We show this imposes an exact, model-free ceiling fixed by how much those realizations agree with each other, and that the ceiling depends on the evaluation metric through a single support-function identity. Measuring assay concordance across four public registries—2,822 MaveDB score sets, 217 ProteinGym assays, two drug screens and 1,150 CRISPR cell lines—we find that two assays of one target agree at 0.56–0.68, and that 541 domains measured twice with different proteases fix assay reliability at 0.897, so 70–90% of every ceiling is irreducible biology rather than noise. Published predictors realize 63% of the achievable on the correlation benchmarks report and 18% on the top-1% selection their users perform. We provide the estimator, the ceilings, and the measurements the field has not made.
Splice-altering variants cause an estimated 15-30% of genetic diseases, yet computational tools lose accuracy outside the canonical GT-AG dinucleotides, leaving intronic variants of uncertain significance (VUS) hard to interpret. Here we present MetaSplice, a 53-feature gradient-boosted ensemble integrating deep-learning splice predictions (SpliceTransformer, Pangolin), evolutionary and gene-level constraint, and splicing-regulatory motifs to score single-nucleotide variants across seven non-exonic splice regions: canonical donor and acceptor sites, donor and acceptor regions, the polypyrimidine tract (PPT), branch point, and deep intronic positions. Trained on 381,226 intronic ClinVar SNVs (35,506 pathogenic), MetaSplice achieved an area under the precision-recall curve (auPRC) of 0.995 in five-fold gene-grouped cross-validation. On a temporally held-out ClinVar test set (107,933 variants), it reached auPRC 0.987 (95% CI 0.985-0.989), outperforming SpliceAI (0.950), CADD v1.7 (0.915), SPIDEX (0.767) and S-CAP (0.253), with the largest gains in the PPT, branch-point and deep intronic regions. As a pathogenicity predictor trained on clinical significance, MetaSplice complements mechanism-specific splice-effect tools. Applied to 47,272 ClinVar splice-region VUS, it nominated 22% for functional follow-up. MetaSplice is freely available as a Docker image.
Xiaoming Liu· bioRxiv· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
