Random Sparse Networks Training with Sharpness-Aware Regularization.

Abstract

Over-parameterization is critical for optimizing neural networks, whereas training sparse networks directly often fails to achieve satisfactory performance. However, the Lottery Ticket Hypothesis (LTH) demonstrates that a randomly initialized dense model has a sparse subnetwork that can be identified through iterative magnitude pruning, while maintaining strong trainability. In this study, we examine a complementary perspective on LTH and propose the Dual Lottery Ticket Hypothesis (DLTH), enabling general training for sparse networks. Specifically, DLTH asserts that any randomly chosen subnetwork drawn from its dense counterparts can be converted to a state with promising trainability. To validate DLTH, we introduce Sharpness-Aware Random Sparse Network Transformation (SA-RST), which uses a regularization term to extrude information from weights slated for masking. Further, this extrusion process is dynamically adjusted based on model sharpness during optimization. Experiments using both CNN and Transformer architectures across several benchmarks demonstrate the effectiveness of our SA-RST and validate the proposed DLTH. Furthermore, we provide comprehensive comparisons with dynamic sparse training methods (RigL, SET) and conduct ablation studies on the sharpness-aware component to validate our design choices. Our study is expected to yield new insights into general sparse network training.