A Self-Supervised Learning Pipeline with Open-Set Detection and Quality Assessment for Embryonic Lineage Classification from scRNA-seq

Abstract

Single-cell RNA sequencing enables high-resolution analysis of early embryonic lineage specification, but most computational pipelines reduce the task to closed-set TE/ICM classification and discard uncertain cells before training. This practice can inflate apparent accuracy while removing the cells most relevant for transition, ambiguity, and out-of-reference analysis. We propose a Self-Supervised Learning (SSL) framework that learns a label-free latent representation, calibrates TE and ICM prototypes from a small trusted reference set, and assigns each cell as known, intermediate, ambiguous, or unknown using an explicit distance-based rule. Seven SSL objectives are evaluated under no-merge and EPI/PrE-to-ICM scenarios with 20/20, 30/30, and 40/40 calibration regimes. On the Meistermann primary cohort of 1751 cells and 2000 highly variable genes, a 10-seed validation shows that reconstruction-based SSL objectives form the strongest group under the composite open-set score, while contrastive-heavy objectives are weaker for unknown detection. The single-seed strict configuration used for detailed analysis is Mask + denoising under no-merge with 30/30 calibration, achieving balanced TE/ICM accuracy of 0.844, unknown AUROC of 0.901, and unknown precision of 0.873. A 10-seed Petropoulos external validation is reported as a dataset-shift stress test. The results support an audit-friendly annotation framework that treats uncertain cells as measurable outputs rather than preprocessing exclusions.