Learning the identity: a case study of how SGD selects among functional decompositions
One might think that learning the identity function with a deep linear residual network is trivial - the path along residual connections already implements the identity, and so the network need only drive its weights to zero. However, this zero-weight solution is just one point on an entire manifold of population-loss...
Andy Arditi, Wei-An Xie, D. Bau et al.
· 0 citations