COMETS: Cost-effective Multi-node Efficient Training System with Memory Pooling and Sharing
Rapid growth of large language models (LLMs) presents major challenges for distributed multi-node training, due to insufficient inter-node bandwidth and lack of scalability in memory expansion. To address both challenges, we propose COMETS, a cost-effective multi-node efficient training system with memory pooling and sharing. By enabling scalable GPU memory expansion and introducing an alternative inter-node data path, COMETS reduces reliance on NIC-based inter-node networking and supports efficient training across diverse cluster environments. We also propose a heterogeneous-aware training strategy to identify optimal training configurations. Experiments show that COMETS improves training throughput by up to 2.11 × over ZeRO-Infinity across eight hardware setups, and boosts performance-per-dollar by 1.98 × and 1.37 × on homogeneous and heterogeneous clusters, respectively. Code is available at: https://github.com/sharc-lab/COMETS.