ZenFlow is presented, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients and achieves up to 5× end-to-end speedup, 2× lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.
Ting-Feng Lan, Yu-Sen Wu, Bin Ma et al.· Proceedings of the ACM on Ma...· 8 citations· ⚡3
The results show that agent-driven resilience engineering is practical when critical state is visible or accessible through coherent abstractions: successful runs finish in under one hour on average, consume about 15M tokens, and produce implementations with negligible failure-free overhead and recovery efficiency comp...
H. Nguyen, Tekin Bicer, Kyle Chard et al.· 0 citations
Agentic artificial intelligence (AI) is moving beyond research demonstrations toward production use at scientific user facilities, including light sources, neutron sources, nanoscience centers, and autonomous laboratories. Its scientific value extends beyond increasing throughput. Agents can perform repeatable tasks in...
Ming Du, Xiang-Yu Yin, Michael H. Prince et al.· 0 citations
Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs si...
Ting-Feng Lan, Yu-Sen Wu, Bin Ma et al.· Proceedings of the ACM on Ma...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.