Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
To scale BP to long contexts, this work introduces context-sharded block parallelism (CSBP), which keeps corrupted K/V and gradients local, avoids replicated clean prefixes, and preserves BDLM training semantics.
Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang et al.
· 0 citations