Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
复制标题
DOI：
--
复制
发表时间：
2018-05
期刊：
影响因子：
--
通讯作者：
Behnam Neyshabur;Zhiyuan Li;Srinadh Bhojanapalli;Yann LeCun;N. Srebro
中科院分区：
文献类型：
--
作者：
Behnam Neyshabur;Zhiyuan Li;Srinadh Bhojanapalli;Yann LeCun;N. Srebro
尽管现有在确保神经网络在规模敏感的复杂性度量方面的概括（例如规范，边缘和清晰度）方面的工作，这些复杂性度量并不能解释为什么神经网络在过度参数化中更好地概括了神经网络。在这项工作中，我们提出了一种基于单位能力的新型复杂度度量，从而导致了两个层Relu网络的更严格的概括。我们的容量结合与测试误差的行为随着网络大小的增加而相关，并可能通过过度参数来解释概括的改善。我们进一步提出了Rademacher复杂性的匹配下限，它比以前的神经网络的容量下限有所改善。
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes, and could potentially explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.