Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 11165-11176· 0 citations· 39 references
TL;DR
Crucially, the analysis reveals that the backbone encoder becomes self-contained post-training: the memory module facilitates the internalization of motif semantics into the model parameters, allowing the backbone to retain performance advantages even when the memory is detached during inference.
Abstract
RNA motifs and Low-Complexity Repeats (LCRs)—recurrent, conserved sequences of nucleotides—serve as the fundamental vocabulary of RNA structure and biological function. While current RNA foundation models have revolutionized sequence modeling through Transformer architectures, they predominantly prioritize modeling global dependencies within the raw sequence. To address this limitation, we propose MotRNA, a motif-aware RNA language model integrating an Explicit N-gram Memory mechanism. Unlike standard implicit neural modeling, MotRNA employs a conditional memory module to explicitly retrieve and fuse motif embeddings based on input k-mers. This mechanism provides direct access to strictly defined local patterns, complementing the global context modeled by self-attention. We validate MotRNA on large-scale RNA datasets. The model achieves a 9.6% absolute improvement in Masked Language Modeling (MLM) accuracy at a 30% masking ratio compared to state-of-the-art baselines. Notably, MotRNA maintains high robustness under extreme masking ratios, indicating effective biological signal reconstruction from sparse context. Crucially, our analysis reveals that the backbone encoder becomes self-contained post-training: the memory module facilitates the internalization of motif semantics into the model parameters, allowing the backbone to retain performance advantages even when the memory is detached during inference.
NucleicBERT is developed, a self-supervised masked-language model that learns contextual representations from single sequences without evolutionary information that advances RNA structure prediction and informs how large language models encode biological information.
Utkarsh Upadhyay, Julian Herold, Markus Götz et al.· Nature Machine Intelligence· 1 citation
ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing, achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction and preserved long-range structural represen...
Ming-Rui Li, Si-Xian Shen, Min-Zhang Li et al.· 0 citations
This work introduces ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments that demonstrates that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded...
Biswajit Banerjee, Claudia A. Carreno, Anton S. Petrov· arXiv.org· 0 citations
SPIRAL is presented, a layer-wise SAE analysis of BiRNA-BERT, a layer-wise SAE analysis of RNA language models where byte-pair tokenization breaks the one-token-one-nucleotide correspondence that nucleotide-level attribution assumes.
M. Hossain, MD. Roqunuzzaman Sojib, Md Toki Tahmid et al.· bioRxiv· 0 citations
RNASeek establishes a unified pretrain–predict–optimize framework that connects learned RNA function to controllable de novo sequence design and provides a general strategy for engineering regulatory RNAs with desired properties.
Shi-Yuan Chen, Neil R. Fernandes, Wei Vivian Li et al.· bioRxiv· 1 citation
RIFT-VAE is best interpreted as a hybrid generative-search framework: pretraining supplies a structure- and family-informed proposal distribution, whereas latent optimization concentrates evaluations in high-scoring regions.