ZOCheck is presented, a fault-tolerant ZO training system that exploits this replayable structure through a CPU shadow process that continuously replays logged updates, materializes consistent recovery images off the GPU critical path, and persists them asynchronously.
Abstract
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress can be represented by lightweight seed-and-scalar step logs, yet naive log-only recovery still incurs replay cost that grows with training progress, and shortcut replay does not preserve the executed floating-point trajectory. We present ZOCheck, a fault-tolerant ZO training system that exploits this replayable structure through a CPU shadow process that continuously replays logged updates, materializes consistent recovery images off the GPU critical path, and persists them asynchronously. ZOCheck therefore combines non-blocking checkpointing during training with fast recovery from a near-current state. We also develop a cost model for choosing the snapshot policy under realistic failure rates. Experiments show that ZOCheck reduces checkpoint overhead by up to 219.7x and recovery latency by 1.55x on average compared with asynchronous full-state checkpointing, translating into up to 21.3x lower end-to-end wasted time across the evaluated failure rates, while preserving exact recovery behavior.
Checkpointing is essential for fault-tolerant large language model training, but it increasingly limits persistence frequency and recovery efficiency as training runs become longer and clusters become larger. This problem is especially acute in Mixture-of-Experts (MoE) systems, where checkpoint bytes are dominated by e...
Guo-Rui Xu, Xiang Liu, Wei-Wei Wu et al.· Proceedings of the 2026 ACM...· 0 citations
This work presents Leto, a fault-tolerant training system that leverages surviving hardware to enable efficient in-place recovery and devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent.
Geon-Woo Kim, Joonseok Kim, Daehyeok Kim· 0 citations
Emerging LLM-powered AI agent workloads, such as test-time tree search and reinforcement learning (RL), require high-frequency state exploration, relying on rapid checkpoint/restore (C/R) of the complete sandbox state, including files and process state. Existing mechanisms duplicate the entire state, causing hundreds o...
Yun-Peng Dong, Jing-Kai He, Shi-Qi Liu et al.· Proceedings of the 2026 ACM...· 22 citations· ⚡1
Solid State Drives (SSDs) are becoming the dominant medium for performance-critical storage. Zoned Namespace (ZNS) SSDs are getting more and more attractive because sequential writes and explicit zone resets reduce address-mapping, over-provisioning, and internal garbage-collection costs. However, reclaiming space requ...
Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a dif...
An-Nan Zhang, Sheng-Jie Song, Wen-Shuai Yao et al.· 0 citations
This work argues that the dominant cost is loss of ready serving capacity, not request progress, so recovery should preserve initialized engine state rather than reconstruct it, and presents fast recovery for Dynamo based on this principle.
Schwinn Saereesitthipitak, Mohammed H. Abdulwahhab, Hannah Zhang et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.