What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes
Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the ques...