CAST: History-Adaptive Spectral Clipping for LLM Pretraining
This work introduces CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC that achieves lower training and mean validation losses than nine gradient-processing baselines, with the fewest relative gradient spikes.
Jia-Cheng Li, Jian-Chao Tan, Ke-Feng Zhang et al.
· 0 citations