K AIROX introduces a Live Pipeline designed to prefetch neurons by predicting next-layer activation patterns, a mechanism that dynamically redistributes neurons between the GPU and CPU based on activation patterns, and a Temporal Activation Momentum cache policy to prioritize neurons with sustained utility while minimizing transient, wasteful transfers.
Deep neural networks (DNNs) with billions of parameters power many important applications, but their training is fundamentally constrained by the limited on-chip memory of GPUs. This memory wall forces training to rely on distributed execution or memory offloading, both of which introduce substantial inefficiencies. Ex...
Xiaoyang Sun, Jie Xu, Zheng Wang· IEEE Transactions on Paralle...· 0 citations
Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs si...
Ting-Feng Lan, Yu-Sen Wu, Bin Ma et al.· Proceedings of the ACM on Ma...· 0 citations
Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-constrained, single-GPU systems. However, this benefit is difficult to realize because expert parameters dominate memory, and token-level routing is dynamic, unpredictab...
NeuroPrefetcher is presented, a storage-backed LLM inference system that exploits that MLP activity during autoregressive decoding has strong temporal locality, and achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.
Nobel Dhar, Md Romyull Islam, Xue-Chen Zhang et al.· Proceedings of the Internati...· 0 citations
LazyTrain is proposed, an optimization layer over a layer-streaming executor that formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training.
Xiao-Jun Wu, Ce-Hao Yang, Hong-Hao Liu et al.· 1 citation
