BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
This paper introduces Batch- Aware Sequence Parallelism (BASP), a sequence parallelism approach that leverages batch structure to reduce communication overhead and localizing communication and improving training efficiency.
B. Ghimire, Jon C. Calhoun
· 0 citations