Learning of deep neural network regression estimates using gradient descent with pruning

Abstract

Estimation of a regression function from independent and identically distributed data is considered. The $L_2$ error with integration with respect to the design variable is used as the error criterion. An initially randomly pruned fully connected deep neural network with logistic squasher as activation function is fitted to the data via gradient descent, using a data-dependent choice of non-constant stepsizes during gradient descent. It is shown that this network achieves (up to a logarithmic factor) the optimal minimax rate of convergence in case that the regression function is $(p,C)$--smooth. Here the estimate is able to circumvent the curse of dimensionality provided the predictors are concentrated in the neighborhood of a low dimensional manifold. The finite sample size performance of the estimate is illustrated by applying it to the simulated data.