Shou-Da Liu

NSP: Accelerating Variable-Length LLM Training via Nested Sequence Parallelism

Long-context LLM training on long-tailed corpora faces a central communication--balance tradeoff. Such sequence-length heterogeneity makes any single sequence-parallelism (SP) degree a poor fit for the workload: a small degree leaves the few long sequences badly imbalanced, while a large degree forces the many short se...