With the rising computational and monetary costs of training large language models (LLMs), checkpointing---periodically storing model states for recovery---becomes essential for fault tolerance. Conventional checkpointing entails a severe trade-off between checkpoint frequency (I/O overhead) and computational recovery...
Min-Qiu Sun, Xin Huang, Luan-Zheng Guo et al.· 1 citation
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress can be represented by lightweight seed-and-scalar step logs, yet naive log-only recovery still incurs replay cost that grows with training...
Min-Qiu Sun, Xin Huang, Luan-Zheng Guo et al.· 0 citations
Hybrid CPU architectures have entered the mainstream desktop computing with the announcement of Intel's Alderlake architecture. Such a transition to heterogeneous CPU architecture has various performance and power implications on existing parallel workloads. In this paper we study the effects and impact of hybrid core...
Nanda Velugoti, Joseph Manzano, Nathan R. Tallent et al.· 0 citations
MEPOWER is proposed, a flexible, model-based approach to exposing compute/data movement imbalance that characterizes the fine-grained memory behavior of parallel workloads that demonstrates a reduction in EDP on a range of HPC benchmarks with minimal impact on execution time when compared to the standard OS/hardware-ma...
Nanda Velugoti, Joseph Manzano, Andrés Márquez et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
