Proceedings of Machine Learning Research vol 125:1{54, 2020 33rd Annual Conference on Learning Theory A Corrective View of Neural Networks: Representation, Memorization and Learning Guy Bresler
[email protected] Department of EECS, MIT Cambridge, MA, USA. Dheeraj Nagaraj
[email protected] Department of EECS, MIT Cambridge, MA, USA. Editors: Jacob Abernethy and Shivani Agarwal Abstract We develop a corrective mechanism for neural network approximation: the total avail- able non-linear units are divided into multiple groups and the first group approx- imates the function under consideration, the second approximates the error in ap- proximation produced by the first group and corrects it, the third group approxi- mates the error produced by the first and second groups together and so on. This technique yields several new representation and learning results for neural networks: 1. Two-layer neural networks in the random features regime (RF) can memorize arbi- Rd ~ n trary labels for n arbitrary points in with O( θ4 ) ReLUs, where θ is the minimum distance between two different points. This bound can be shown to be optimal in n up to logarithmic factors. 2. Two-layer neural networks with ReLUs and smoothed ReLUs can represent functions with an error of at most with O(C(a; d)−1=(a+1)) units for a 2 N [ f0g when the function has Θ(ad) bounded derivatives. In certain cases d can be replaced with effective dimension q d. Our results indicate that neural networks with only a single nonlinear layer are surprisingly powerful with regards to representation, and show that in contrast to what is suggested in recent work, depth is not needed in order to represent highly smooth functions.