Concord: Building Consensus Representations for Single Cells with Collaborative Random Projection

Abstract

Foundation models (FMs) have recently transformed single-cell genomics by learning transferable representations from large-scale single-cell data, enabling a wide range of downstream biomedical applications. Inspired by natural language processing, existing single-cell FMs adapt transformer architectures by treating genes as tokens and cells as sequences. However, transformers are inherently agnostic to input order, while genes lack a natural sequential structure. Current approaches rely on heuristic strategies, such as expression-based gene sorting, to impose positional information, which often fail to capture relative relationships among collectively expressed genes and between gene identities and their expression levels, leading to information loss and limited generalization. In this work, we propose Concord, a novel single-cell FM that explicitly models relationships between gene identities and expression levels through a collaborative rotary attention (CRA) mechanism. Specifically, Concord employs two collaborative attention modes: a gene-to-expression rotational attention that produces gene-enhanced expression representations, and an expression-to-gene rotational attention that yields expression-enhanced gene representations. These two processes provide distinct yet complementary views of the same cell-level expression profile; accordingly, we employ contrastive learning to align their semantic representations. Our theoretical analysis demonstrates that CRA effectively leverages the relative distances in one embedding space to establish stable and consistent dependencies in the other modality. Extensive experiments on various single-cell datasets demonstrate that Concord outperforms existing FMs across various downstream tasks and provides more informative and transferable gene and cell representations. Our code is available at https://github.com/Catchxu/Concord.