Covariance-Aware Newton-Schulz Orthogonalization for Noise-Robust Sequential Recommendation

Abstract

Autoregressive sequential recommendation trains transformer models on tokenized user behavior via cross-entropy. The Muon optimizer projects gradient matrices onto the Stiefel manifold through Newton-Schulz (NS) orthogonalization, ensuring uniform utilization of all parameter directions. However, we identify that this projection equalizes all gradient singular values to unity, collapsing the natural energy separation between signal and noise subspaces. In recommendation, where long-tail distributions, noisy implicit feedback, and exposure bias produce corrupted gradients, this equalization promotes noise directions to the same magnitude as signal directions—a directional amplification that magnitude-based techniques such as gradient clipping and sharpness-aware minimization (SAM) cannot address. We propose Covariance-Aware Newton-Schulz (CovNS), which estimates gradient directional stability via an exponential moving average (EMA) of the gradient outer product and attenuates volatile directions before orthogonalization. Experiments on a public benchmark and an industrial dataset show that CovNS outperforms Muon and all baselines across all retrieval metrics on both datasets with minimal overhead.