NucleicBERT interprets RNA sequence space through self-supervised language modelling
TL;DR
NucleicBERT is developed, a self-supervised masked-language model that learns contextual representations from single sequences without evolutionary information that advances RNA structure prediction and informs how large language models encode biological information.
Abstract
Much of the human genome’s non-protein-coding fraction acts directly through RNA, yet the structural and functional roles encoded in these sequences remain poorly understood. Applying deep learning is hindered by scarce RNA structural data and it remains unclear what biological constraints such models can recover directly from the abundant RNA sequences alone. Here, to address these challenges, we developed NucleicBERT, a self-supervised masked-language model that learns contextual representations from single sequences without evolutionary information. Explainable artificial intelligence analyses show that the model organizes RNA sequences in latent space and encodes structural properties indicating that biologically meaningful constraints are learned from sequence correlations alone. When fine-tuned for downstream structural and functional tasks, NucleicBERT requires only single sequences while matching or exceeding current RNA prediction models. This alignment-free framework addresses the scarcity of annotated 3D RNA data while providing a rapid, computational complement to experimental techniques. By bridging abundant unlabelled sequence data with scarce structural annotations, NucleicBERT advances RNA structure prediction and informs how large language models encode biological information. RNA structure and function are hard to infer because annotations are scarce, despite abundant sequence data. Upadhyay et al. trained a self-supervised model on large-scale RNA data that derives biologically meaningful patterns from sequence correlations.