A Survey: Lossless Model Accuracy Data Partitioning and Training Parallelism Strategies for Distributed Machine Learning

Abstract

In the digital era of big data and large-scale models, conventional stand-alone computing power and storage approach fails to meet the demands of large-scale machine learning models, therefore, distributed data parallelism has been the key supporting techn ology for breaking hardware bottleneck. However, current distributed training is facing numerous actual conflicts in the practical applications: data partitioning and resource scheduling, heterogeneous clusters and global synchronization. This passage elaborates on the practical dilemmas of distributed machine learning training, achieving thorough integration of data partitioning patterns, parallel operation architectures and precision stabilization strategies by three dimensions: problems, methods and findings. The study points out that existi ng technology manifests multiple limitations, such as inefficient training iteration, poor flexibility of static partitioning, and the difficulty in simultaneously maintaining accuracy invariance and convergence consistency. Subsequent research can develop with the direction of building Non -Independent and Identically Distributed (Non -IID) adaptive partitioning mechanisms, constructing efficient resource coordination platforms and correcting local gradients, which provides reliable references for establishing high-precision distributed machine learning systems.