Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling

Abstract

Pruning network connections is key to brain function but, despite its importance, there exist few biologically-plausible pruning rules with demonstrated good performance. We evaluate noise-prune, a recently introduced unsupervised local pruning rule for recurrent networks that uses noisy fluctuations to determine the importance of connections. Noise-prune has previously only been empirically tested on random networks without a specific computational function. We show that noise-prune preserves task-performance in task-trained recurrent neural networks, greatly outperforming a strategy that only uses connection magnitude and exceeding a non-local strategy that uses second-order information. Rather than deterministically removing connections that fall below a threshold, noise-prune samples connections to preserve based on their importance and strengthens retained connections to preserve average strength. We show that this sampling and rescaling is essential to good performance, and can augment other pruning strategies. We also find that the optimal empirical degree of rescaling is lower than predicted by the original theoretical argument, and thus develop a new version of noise-prune that outperforms the previous version. Our results demonstrate that a biologically-motivated pruning strategy based on local activity statistics can effectively preserve task performance in functional recurrent networks by combining activity-dependent connection selection with compensatory strengthening of surviving connections.