When Data Augmentation Improves the Calibration and Accuracy of Deep Neural Networks

Abstract

Accurately quantifying the predictive confidence or improving model calibration has recently attracted increasing attention, in contrast to focusing only on maximizing model accuracy. This is essential especially for safety-sensitive applications. Among various techniques for calibration improvement, some data augmentation methods such as mixup and CutMix have been shown to improve the calibration and accuracy of deep learning models of certain architecture families. However, the effect of a broader range of data augmentation methods and models as well as the interplay between the complexity of the machine learning problem and the extent of calibration improvement using data augmentation needs to be further explored. The novel aspect of this study is the investigation of how different data augmentation techniques affect the calibration and accuracy of different families of neural networks in image classification tasks, across datasets with varying class counts. In this paper, neural networks are trained on datasets with differing class counts under two experimental conditions: with and without data augmentation, and differences in the accuracy, expected calibration error (ECE) and negative log likelihood (NLL) in both scenarios are computed and compared. The experiments confirm that different data augmentation methods consistently improve the calibration across datasets with varying class counts. Furthermore, the findings suggest that the extent of calibration and accuracy improvements by using a mixing data augmentation method such as mixup and CutMix is higher than by using an automated augmentation method such as RandAugment or trivialaugment. This indicates that mixing data augmentation methods tend to have a more pronounced impact on improving calibration and accuracy for classification tasks.