ARCHER-LD: Rapid Long-Range Linkage Disequilibrium Calculations at Biobank Scale using GPU Acceleration

Abstract

Linkage disequilibrium (LD) information from one’s own dataset is considered optimal for downstream analyses such as statistical fine-mapping. However, the computational complexity ( computations for N variants) lead most studies to use external reference panels, such as from 1000 Genomes. To capture LD across all variant pairs in biobank-scale whole-genome sequencing (WGS) datasets with hundreds of millions of variants, new computational strategies are essential. We present a novel approach that uses multi-GPU distributed computing to compute R2 for every variant pair in a dataset. On chromosome 22 (1.8 million variants) of the 30x WGS 1000 Genomes dataset, our method using eight consumer-level 12GB GPUs (NVIDIA RTX 2080TIs) takes <10 minutes, while the same calculation with PLINK using a high-end 64- threaded CPU (Intel Xeon Gold 6338) takes >80 minutes, corresponding to a ≈8x speedup. We further demonstrate true biobank-scale performance in the Penn Medicine Biobank (PMBB; 57,170 samples), computing chromosome 1 LD (≈1.38 million variants) in 49 minutes versus 22.7 hours for PLINK, a ≈28x speedup. With this method, we successfully computed on the aforementioned 30x WGS 1000 Genomes dataset (≈120 million variants and ≈2500 samples) the entire LD matrix (>1e16, or 10 quadrillion elements) in under 6 hours using 512 NVIDIA 40GB A100 GPUs on the Department of Energy Argonne Leadership Computing Facility Polaris Supercomputer. We make this tool, coded in Python using CuPy, publicly available. Using this tool, researchers can leverage the full extent of their genomic data without relying on external LD reference panels and acquire more accurate, population-specific findings, particularly for groups underrepresented in existing databases. Key Messages ARCHER-LD makes use of GPUs to speed up the computation of linkage disequilibrium dramatically. After conversion of genotype data to a disk-based matrix, we can compute LD in parallel over chunks of the genotype matrices for both phased and unphased data. By making use of a parallel formulation of LD calculations, ARCHER-LD can scale to any number of GPUs that are available. ARCHER-LD scales to real biobanks: in the Penn Medicine Biobank (57,170 samples) it computes whole-chromosome LD substantially faster than PLINK, with the speedup increasing as the dataset grows. ARCHER-LD will be publicly available for use in calculating in-sample LD and extension by other researchers.