Alignment-Aware Genomic Coordinate Encoding for Long-Range DNA Modeling

Abstract

Current genomic foundation models (GFMs) largely inherit the language-modeling paradigm over contiguous token sequences, treating DNA as ordinary text. The genome, however, is a coordinate-based system of sparse signals: the entities that matter, such as individual variants, regulatory elements, and distal gene interactions, occupy a vanishing fraction of chromosome-scale sequences and are defined by their positions in the reference coordinate system rather than by linear proximity. This mismatch limits pure DNA-sequence modeling on individualized and non-contiguous genomic tasks, where inputs are naturally variant-centric and sparsely sampled. We present USM-AA, a genomic foundation model built on two complementary components: an alignment embedding module (AEM) that represents alignment information, capturing each token’s chromosome, strand, and comparison outcome against the reference so that variant signals are explicitly represented; and an Alignment-Aware Rotary Position Embedding (AARoPE) that performs non-contiguous positional encoding, placing tokens on the reference coordinate system so that the model maintains correct positional relationships over non-contiguous input and supports end-to-end training. The model is further trained with a reverse-complement masked learning strategy that operates purely at the loss level, adding no parameters and requiring no architectural modification. Pretrained on large-scale genomic sequences spanning cross-species evolution and human population variation, USM-AA achieves state-of-the-art performance across multiple genomic tasks, including long-range regulatory prediction and variant-effect prediction. The source code is publicly available at https://github.com/Jimmy0743/USM-AA.