Skip to content

ZOCheck: CPU-Shadow Checkpointing for Zeroth-Order LLM Fine-Tuning

Sep 2026 · arXiv.org · Vol abs/2609.27189 · 1 citation · 48 references
Computer Science

TL;DR

ZOCheck is presented, a fault-tolerant ZO training system that exploits this replayable structure through a CPU shadow process that continuously replays logged updates, materializes consistent recovery images off the GPU critical path, and persists them asynchronously.

Abstract

Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress can be represented by lightweight seed-and-scalar step logs, yet naive log-only recovery still incurs replay cost that grows with training progress, and shortcut replay does not preserve the executed floating-point trajectory. We present ZOCheck, a fault-tolerant ZO training system that exploits this replayable structure through a CPU shadow process that continuously replays logged updates, materializes consistent recovery images off the GPU critical path, and persists them asynchronously. ZOCheck therefore combines non-blocking checkpointing during training with fast recovery from a near-current state. We also develop a cost model for choosing the snapshot policy under realistic failure rates. Experiments show that ZOCheck reduces checkpoint overhead by up to 219.7x and recovery latency by 1.55x on average compared with asynchronous full-state checkpointing, translating into up to 21.3x lower end-to-end wasted time across the evaluated failure rates, while preserving exact recovery behavior.

View source

Similar papers

Book Open access Nov 2026

MoECatch: Efficient Checkpointing and Consistent Recovery for Mixture-of-Experts Models

Checkpointing is essential for fault-tolerant large language model training, but it increasingly limits persistence frequency and recovery efficiency as training runs become longer and clusters become larger. This problem is especially acute in Mixture-of-Experts (MoE) systems, where checkpoint bytes are dominated by e...

Guo-Rui Xu, Xiang Liu, Wei-Wei Wu et al. · 0 citations
#machine learning Preprint Sep 2026

Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

This work presents Leto, a fault-tolerant training system that leverages surviving hardware to enable efficient in-place recovery and devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent.

Geon-Woo Kim, Joonseok Kim, Daehyeok Kim · 0 citations
#artificial intelligence Book Open access May 2026

DeltaBox: Millisecond-Scale Sandbox Checkpoint and Rollback for Stateful AI Agents

Emerging LLM-powered AI agent workloads, such as test-time tree search and reinforcement learning (RL), require high-frequency state exploration, relying on rapid checkpoint/restore (C/R) of the complete sandbox state, including files and process state. Existing mechanisms duplicate the entire state, causing hundreds o...

Yun-Peng Dong, Jing-Kai He, Shi-Qi Liu et al. · 22 citations · ⚡1
Preprint Oct 2026

RetainZ: Reclaim-Time-Aware Placement for AI Checkpoints on Zoned SSDs

Solid State Drives (SSDs) are becoming the dominant medium for performance-critical storage. Zoned Namespace (ZNS) SSDs are getting more and more attractive because sequential writes and explicit zone resets reduce address-mapping, over-provisioning, and internal garbage-collection costs. However, reclaiming space requ...

Min-Xing Chu, Ruo-Xin Yang · 0 citations
Preprint Oct 2026

State-Aware Activation Recomputation with Replay Contracts and Selective Preservation

Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass saw. Mutable buffers, aliased bytes and random-generator state can change in between, so re-executing unchanged code can differentiate a dif...

An-Nan Zhang, Sheng-Jie Song, Wen-Shuai Yao et al. · 0 citations
Sep 2026

Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo

This work argues that the dominant cost is loss of ready serving capacity, not request progress, so recovery should preserve initialized engine state rather than reconstruct it, and presents fast recovery for Dynamo based on this principle.

Schwinn Saereesitthipitak, Mohammed H. Abdulwahhab, Hannah Zhang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.