Preprint
Oct 2026
Fast and Memory Efficient Offload Training Framework with Hybrid XPU Computation
With the ever-growing size of deep learning models, GPU memory is prone to being insufficient during training. A prominent approach is ZeRO-Offload, which moves the optimizer states to CPU memory and performs parameter update using CPU. However, the deficiencies of ZeRO-Offload include low GPU utilization, imperfect ov...
Zhi-Yi Yao, Zu-Ning Liang, Yue-Dong Xu et al.
· 0 citations