Saliency-Informed Sparsity For Gating Deep Convolutional Feature Space

Abstract

Biological vision has evolved to make efficient use of the limited information processing capability and tight energy budget of the brain by preferentially processing the most salient features of visual scenes. In contrast, modern deep vision models rely on expansive, high-dimensional representations. This may offer potential recognition gains but increases computing costs. As a consequence, the computer vision community has been exploring sparsity-enforcing techniques such as activation dropout and weight pruning. Beyond ameliorating the burden of computation, sparsity techniques such as random dropout have been shown to regularize model training, thus allowing for better generalization. Here, we pursue both dropout and weight pruning in tandem and adopt a biologically plausible saliency-informed dropout technique as an explainable alternative to the unstructured standard random dropout approach. Our approach involves a hierarchical “retinotopic" gating of convolutional feature maps, which promotes efficient deletion of “redundant" weights by iterative magnitude pruning. We explore the effectiveness of saliency-informed dropout based on different approaches to dropping based on the saliency map, dropout through all layers or only early layers, and by additionally applying dropout at inference. We compare throughout with standard random dropout. We present empirical results on regimes where such structured dynamic and static (weight) sparsities interact optimally to prune a ResNet model on the Imagenette dataset.