Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction

Abstract

Single-cell foundation models (scFMs) are commonly benchmarked using final-layer embeddings, although useful depth may depend on the task. We evaluate Geneformer, scBERT, scGPT, and scFoundation as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses. Intermediate layers significantly improve cell-type accuracy for scGPT and scBERT in several settings, while Geneformer and scFoundation generally show performance comparable to their final layers. For perturbation prediction, LODO identifies layer 1 as an earliest-on-par candidate across all four models, without establishing universal superiority. Paired scGPT CLS and mean-pooling analyses show that readout affects preferred depth. CKA and representation metrics reveal model- and task-dependent geometric changes, while retrieval and collapse checks support the robustness of perturbation findings. We recommend reporting extraction depth, readout, selection protocol, and task-specific diagnostics. A validated non-final layer may also reduce computation, although efficiency was not directly benchmarked.