Geometry of Optimization and Implicit Regularization in Deep Learning
Geometry of Optimization and Implicit Regularization in Deep Learning
复制标题
DOI：
--
复制
发表时间：
2017-05
期刊：
影响因子：
--
通讯作者：
Behnam Neyshabur;Ryota Tomioka;R. Salakhutdinov;N. Srebro
中科院分区：
文献类型：
--
作者：
Behnam Neyshabur;Ryota Tomioka;R. Salakhutdinov;N. Srebro
我们认为，优化通过隐式正则化在深度学习模型的泛化中发挥着至关重要的作用。我们通过证明泛化能力不是由网络大小控制而是由其他一些隐式控制控制来做到这一点。然后，我们演示如何改变经验优化程序可以提高泛化能力，即使实际优化质量不受影响。我们通过研究深度网络参数空间的几何形状并设计适合该几何形状的优化算法来实现这一点。
We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.