GenVarLoader, an accelerated dataloader that creates personalized genomic sequences and functional tracks on-the-fly to overcome severe data loading bottlenecks, provides a scalable solution for integrating biobank-scale personalized genomic data with deep learning applications.
Abstract
Deep learning models can predict the functional impact of genetic variants. However, training and applying these sequence models to personalized genome sequences are severely constrained by the computational burden of storing and reading massive datasets.
We developed GenVarLoader, an accelerated dataloader that creates personalized genomic sequences and functional tracks on-the-fly to overcome these bottlenecks. GenVarLoader stores this personalized genomic data in formats specifically optimized for machine learning. Our tool reads variants up to 26,000 times faster than a pre-subset BCF, loads personalized data up to 130 times faster than tuned FASTA and pyBigWig pipelines, and reduces storage approximately 2,000-fold compared to existing alternatives.
By eliminating severe data loading bottlenecks, GenVarLoader ensures efficient model training and inference. It provides a scalable solution for integrating biobank-scale personalized genomic data with deep learning applications.
Linkage disequilibrium (LD) information from one’s own dataset is considered optimal for downstream analyses such as statistical fine-mapping. However, the computational complexity ( computations for N variants) lead most studies to use external reference panels, such as from 1000 Genomes. To capture LD across all vari...
Rachit Kumar, Pankhuri Singhal, David Y. Zhang et al.· bioRxiv· 0 citations
FP8 largely preserves biological fidelity across the evaluated scales and inference regimes, while reducing GPU memory footprint at 4B scale and improving energy efficiency during autoregressive generation, however, realized throughput gains remain substantially below FP8’s theoretical 2× hardware ceiling.
Mutian Yu, Robert Egan, Feng-Chen Liu et al.· bioRxiv· 0 citations
Summary Protein mutation datasets are widely used in variant effect prediction and protein engineering, but datasets from different sources often lack consistent conventions for mutation representation, sequence representation, data organization, and experimental labels, making these resources difficult to integrate di...
Zi-Yu Shi, Yu-Xiang Tang, Meng-Xin Yang et al.· bioRxiv· 0 citations
Rapid medical triage following ionizing radiation exposure is critical for emergency management, yet traditional alignment-based bioinformatics are too computationally intensive for mass-casualty scenarios. To address this, we developed STUART (Sequence Triage and qUAntification of Read Transcripts), a mapping-free mac...
Tomasz Strzoda, Lourdes Cruz-Garcia, Mustafa Najim et al.· 0 citations
GB.GeneUnet, an 837M-parameter transformer-based U-Net pretrained on 6 trillion tokens from multi-species genomes in OpenGenome2 is introduced, extending genomic context to 1 Mb with up to 100× inference speedup over GeneMoE, a preliminary MoE transformer baseline of similar model size pretrained on the same data.
Ning Sun, William de Vazelhes, Pan Li et al.· bioRxiv· 0 citations
Population-scale sequencing now produces variant call sets with thousands of samples and millions of sites, making post-calling analysis a recurring bottleneck. Because the Variant Call Format stores every field of every record together, a tool answering a field-limited question still parses the unused annotations, FOR...
E. Estaji, Jian-Feng Mao· bioRxiv· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
