A Comparative Study of Optimization Algorithms, Regularization Techniques, and CNN Architectures for CIFAR-10 Image Classification

Abstract

In this paper, we perform a systematic empirical study of deep learning techniques on the CIFAR-10 image classification benchmark. We study three inter-related aspects of neural network design: (1) the relative impact of nine gradient descent optimisation algorithms on a baseline Multi-Layer Perceptron (MLP); (2) the effectiveness of six regularisation techniques namely; L2 weight decay, Dropout, Data Augmentation, Gaussian Noise Injection, Early Stopping, and Ensemble Learning; on model generalisation; and (3) a benchmark comparison of five landmark Convolutional Neural Network (CNN) architectures alongside Inception V3 trained from scratch. We also investigate autoencoder-based image reconstruction and a side-by-side comparison of linear Principal Component Analysis (PCA) with non-linear autoencoder representations. Nesterov Accelerated Gradient achieves the lowest training loss (1.5814) in short-run experiments. VGGNet achieves the highest test accuracy (75.09%) among CNN architectures trained from scratch for five epochs. Early Stopping yields the best regularisation outcome, and non-linear autoencoders outperform PCA in reconstruction quality. All experiments use TensorFlow 2.x and Keras on CIFAR-10.