Jul 2026
Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool
Long-context LLM training suffers from a load-balancing problem that sequence packing does not solve. Packing samples into fixed-token sequences balances memory and linear-cost operators, but the dominant attention cost scales with the sum of squared sequence lengths. Thus, equally sized packed sequences drawn from a l...
Yan Wang, Xiu-Long Yuan, Kaiming Yang et al.
· arXiv.org · 1 citation