Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
This paper proposes DPP, which transforms PP granularity from a static design choice into a workload-adaptive optimization space over packed, split, and hybrid chunks and further introduces a new coupling between heterogeneous pipeline scheduling and gradient checkpointing.
Shiju Wang, Yujie Wang, Fang-Cheng Fu et al.
· 0 citations