Comparative Benchmarking of Probabilistic, Recurrent, and Self-Attention Models for Autoregressive Genomic Sequence Modeling
TL;DR
This work provides a resource-efficiency analysis and empirical benchmark for model selection in constrained genomic modeling tasks, highlighting that computationally expensive deep learning architectures do not unconditionally outperform lightweight statistical baselines on small, vocabulary-constrained genomic datasets, and identifies clear directions for future investigation.
Abstract
Transformer-based language models have achieved remarkable success in natural language processing, and their structural parallels with DNA sequences, both being linear strings over a finite alphabet, motivate their application to genomics. Although discriminative genomic language models such as DNABERT have been explored, autoregressive generative approaches remain comparatively underutilized. This study presents an empirical comparative evaluation of three classes of autoregressive sequence models: N -gram statistical models, long short-term memory (LSTM) recurrent networks, and transformer-based architectures, applied to human gene nucleotide sequences. Rather than processing full-length genomic sequences, which impose prohibitive computational costs, we restrict analysis to sequences of up to 1,000 nucleotides sourced from the National Center for Biotechnology Information Gene Database. Models are evaluated using perplexity on held-out sequences and, more practically, by their ability to distinguish genuine gene sequences from synthetically mutated variants across three mutation levels. Our results demonstrate that LSTM-based models consistently achieve the best mutation-detection accuracy across all conditions, while N -gram models with Laplace smoothing perform competitively relative to their simplicity and low computational cost. Transformer models, despite their theoretical capacity for long-range dependency modeling, show lower mutation-detection accuracy in this constrained, short-sequence setting. This work provides a resource-efficiency analysis and empirical benchmark for model selection in constrained genomic modeling tasks. It highlights that computationally expensive deep learning architectures do not unconditionally outperform lightweight statistical baselines on small, vocabulary-constrained genomic datasets, and identifies clear directions for future investigation.