In-Copy Fusion: Runtime Argument Fusion for Efficient OpenMP GPU Offloading

Abstract

In modern GPU-based non-unified heterogeneous systems, CPU-GPU communication happens via the PCI bus. Data transfers are affected by startup overhead, which underutilizes the PCI channel bandwidth for small transfers. Modern programming models, such as CUDA and OpenMP, treat each input argument to a compute kernel independently, leading to data movement segmentation and execution slowdowns. This work presents In-Copy Fusion (ICF), a runtime optimization of OpenMP’s accelerator model, implemented in LLVM, that gathers eligible map clauses into staging buffers, orchestrating an optimal transfer pipeline through fusion without modifying application or kernel code. The technique preserves OpenMP semantics with negligible overhead. We evaluate ICF across a range of representative HPC benchmarks and configurations, including varying argument counts, data sizes, and argument-size disparities, as well as real-world benchmarks. The results show that ICF improves effective host-to-device (H2D) bandwidth and reduces end-to-end time relative to the baseline runtime with per-argument transfers across platforms and workloads, achieving an up to 4.8 × speedup.