A pre-registered, final-test-once evaluation framework that locks the outcome rule, seeds, and target-gene-grouped splits before any test data are seen, and scores each frozen representation against a strong expression baseline, a matched-capacity Gaussian control, and a within-split row-identity (shuffle) control.
Abstract
Transcriptomic foundation models are increasingly used as reusable cell and gene representations, but validating them on new data under weak supervision and distribution shift is hard: standard comparisons conflate genuine representation signal with model capacity, row-identity artifacts, gains over strong task-specific baselines, and outcome rules chosen after seeing the test set. We introduce a pre-registered, final-test-once evaluation framework that locks the outcome rule, seeds, and target-gene-grouped splits before any test data are seen, and scores each frozen representation against a strong expression baseline, a matched-capacity Gaussian control, and a within-split row-identity (shuffle) control; only the per-cell embedding-extraction step is model-specific. Applying it to three architecturally distinct models-Geneformer, scGPT, and UCE-across two external Replogle Perturb-seq datasets (RPE1 and K562), all three clear the capacity and row-identity controls by a wide margin, yet none reliably beats the expression baseline: the strongest (Geneformer) exceeds it by at most about $0.03$ test $R^2$ and clears the pre-registered four-of-five-seed threshold in neither dataset, while scGPT and UCE fall below it. All three therefore land in the same pre-registered partial-replication category-a consistent cross-architecture outcome, even though the baseline-relative gap differs in sign and magnitude across models. These representations carry real structure beyond trivial controls but, under this weak magnitude label, do not transfer past a simple strong baseline; the locked framework is reusable for any frozen transcriptomic representation by swapping only the extraction step.
A representation-accessibility analysis of frozen genomic language models across regulatory, epigenetic, promoter, splice-site, and variant-effect prediction tasks shows that local biological signal is partially present in frozen representations, but is not always accessible through final pooled embeddings.
Nirjhor Datta, Swakkhar Shatabda, M. S. Rahman· 0 citations
Large-scale single-cell perturbation atlases make it possible to ask an inverse question: given an observed transcriptional response, which annotated targets and compounds in a fixed library are most consistent with that response? We present \model, a Transformer retrieval model for this closed-library setting. Each input is a cell-level perturbation signature formed by contrasting one treated cell with a cell-line-specific mean DMSO reference. The encoder maps the signature to a target-retrieval vector and a molecular-embedding vector, trained jointly with supervised target losses and structure--transcriptome alignment. We evaluate on Tahoe-100M conditions with mapped target annotations using a within-compound stratified 90/10 condition-pair split of 10,505 training and 1,168 validation drug--cell-line pairs. Because compounds and cell lines can occur in both partitions, the experiment measures held-out condition-pair retrieval rather than generalization to unseen compounds or cellular contexts. In a Monte Carlo evaluation over 38,400 sampled validation cells, \model\ achieved target Recall@10 of 0.408 and Recall@20 of 0.544, together with compound Hit@1 of 0.129, Hit@10 of 0.343, and mean reciprocal rank of 0.205 over a 379-compound bank. A separate diagnostic evaluation produced nearly identical values for the main model and large gains over a random-vector control and post-hoc bag-of-genes controls. These results demonstrate that a single multi-task model can recover both mapped target annotations and recorded compound identities from observed cell-level responses in the evaluated Tahoe-100M closed-library setting. Generalization to unseen compounds and cellular contexts remains to be established.
Single-cell and spatial foundation models promise transferable biological representations, yet their generality remains largely untested across modalities, biological domains and analytical tasks. We benchmarked six representative models, Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo and Novae, using a harmonised framework spanning scRNA-seq, spatial transcriptomics and Perturb-seq. We evaluated zero-shot and continually pretrained clustering, supervised annotation, marker-gene concordance and perturbation prediction. Model performance was strongly conditional: expression-trained cell-level transformers best resolved many cell-identity tasks, spatial and graph-aware models better preserved tissue architecture, and language-derived gene embeddings were competitive for selected perturbation-response metrics. No model dominated across tasks, and rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice. This benchmark provides practical guidance for model selection and argues that future models should be judged by biological generalisation, interpretability and perturbation-grounded validity, not by scale or leaderboard performance alone.
Sally Chen, Roxana Zahedi, Lucy Chhuo et al.· 0 citations
Open-weight language models are fine-tuned, quantized, pruned, and merged, yet their provenance is often undocumented. We study data-free white-box lineage verification: can weights alone reveal whether two compatible model checkpoints share ancestry? Residual training produces a shared identity-aligned component in branch products, so this structure alone cannot establish ancestry. We remove it and compare checkpoint-specific structure across residual blocks, yielding a symmetric lineage score calibrated against independent checkpoints. On residual-MLP and GPT-2 benchmarks, the score separates fine-tuned, LoRA-merged, pruned, and quantized descendants from independent and distilled models (AUROC=1.0), distinguishing weight ancestry from behavioral similarity. Under function-preserving checkpoint laundering experiments, weight-space baselines lose margin or fail; our score remains unchanged and runs 76x faster than the nearest robust baseline on GPT-2. The projection-pairing signal appears across six language-model families and beyond, and a case study correctly identifies 3 related and 7 unrelated LLaMA-2 public checkpoints. Collectively, these results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints
A discovered sub-population from single-cell transcriptomic data is only meaningful if it is reproducible, yet clustering is usually done with one method on one embedding and rarely tested. We present a label- free, multi-metric pipeline that reframes clustering as an auditable, methods-blind decision and separates two notions of stability that are commonly conflated: reproducibility under cell resampling (bootstrap) and reproducibility under re-embedding (retraining the representation). The pipeline evaluates seven clustering configurations across cluster counts using five non-redundant quality metrics. As a whole- dataset control on a mouse retinal atlas, it recovers an eight-cell-type annotation at 96.3% accuracy (adjusted Rand index, ARI = 0.91) without labels. We then validate the discovery mode on two cell types with opposite ground truth. On bipolar cells, which have well-established subtypes, the pipeline accepts the sub-structure: across-embedding reproducibility rises with cluster number to a high plateau (mean pairwise ARI ∼0.93 near the ∼15 known bipolar subtypes), with quality metrics improving in parallel. On rod photoreceptors, treated as homogeneous, it rejects over-clustering: the metric-selected partition passes a bootstrap-stability check but is not reproducible when the embedding is retrained (mean pairwise ARI = 0.69), and the metrics do not improve with cluster number. On synthetic data, the test recovers real structure down to a 5% subpopulation while rejecting null data (high sensitivity and specificity). Bootstrap stability alone is therefore insufficient evidence for sub-population; the across-embedding test discriminates real sub-structure from over-clustering and applies to any cell type as a reproducible alternative to single-method, single-embedding clustering.
Zachary Yousef, Jonah Simone, Dylan Klein et al.· bioRxiv· 0 citations
Cellular state transitions underlie adaptation, ageing and disease, yet prioritizing catalogued genetic perturbations whose expression signatures match an observed transcriptomic shift remains difficult. Most models predict phenotype from a nominated intervention, whereas genetic inverse benchmarks are largely restricted to within-screen identity recovery. Here we introduce RevPert, a gallery-native reverse perturbation model that ranks a fixed genetic catalog for a query contrast ΔY ⋆ = YB − YA by combining signed Pearson connectivity with a learned residual. Across Replogle Essential Perturb-seq (four lines) and LINCS-KO screens (ten lines), RevPert recovered held-out interventions at leading performance relative to matched baselines. Applied to public drug-resistance contrasts in HCC and CML, dual-arm ranking placed pre-specified disease anchors far higher on the expected arms than ranking the same signatures by differential-expression magnitude alone (Essential residual model for HCC; a transductive GWPS residual for CML). RevPert therefore couples within-screen reverse ranking to a screen-external signed-geometry check; the latter calibrates literature anchors and is not claimed as held-out recovery.
Shiyang Liang, Chen Yang, Jingjie Wang et al.· bioRxiv· 0 citations