Joongun Park

Closing the Efficiency Gap: AI Datacenter Co-design Roadmap for Scalable Training of LLMs

This work introduces Calculon-MoE, a tool that jointly explores FLOPS, HBM bandwidth and capacity, multiple network topologies, and popular parallelism/optimization strategies used in LLMs, and quantifies the benefits of overlapping compute and communication, hardware-accelerated collectives, widening the scale-up domain, and higher memory bandwidth and capacity.