DiCTR: A Distributed Lightweight Second-Order Optimization Algorithm for Deep Model Training

Abstract

Optimization algorithms, also referred to as optimizers, play a critical role in accelerating deep neural network training. First-order optimizers based on stochastic gradients are widely adopted in fields such as computer vision and natural language processing due to their ease of implementation and low computational cost. However, their theoretical convergence rate remains unsatisfactory. In contrast, second-order optimizers, which leverage the Hessian or Fisher information matrix to refine gradient updates, are emerging as promising alternatives to first-order methods. Unfortunately, classic second-order optimizers face significant challenges in both computation and memory. In this paper, we propose a second-order optimizer named distributed Cholesky triangular replacement (DiCTR). It makes two main contributions: (1) it introduces a lightweight second-order method that employs efficient matrix factorization and sparse matrix replacement techniques to reduce system computation and memory overhead; (2) it designs a novel distributed training framework encompassing both data parallelism and model parallelism, utilizing more efficient communication primitives instead of unnecessary AllReduce operations. We evaluate DiCTR’s performance by training representative models such as ResNet and BERT on real-world datasets. Experimental results demonstrate that our algorithm outperforms the baseline methods. For instance, DiCTR achieves a 1.13 × to 2.64 × improvement in training time on the classic ResNet-20 model, and a 1.29 × to 1.45 × improvement on the BERT‑large model. Moreover, compared to the baseline methods, DiCTR reduces memory usage by at least 14.18%.