The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program
The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Yifei Wang;Mert Pilanci
中科院分区：
文献类型：
--
作者：
Yifei Wang;Mert Pilanci
我们从凸几何和对偶的角度研究非凸次梯度流用于训练两层ReLU神经网络。我们将非正则化非凸梯度流的隐偏差描述为等效凸模型的凸正则化。在此凸优化问题中，我们证明了非凸次梯度流的极限点可以通过原对偶对应来识别。此外，我们在对偶变量上导出了保证非凸目标的平稳点是凸目标的KKT点的充分条件，从而证明了非凸梯度流收敛于全局最优。对于一类正则训练数据分布，如正交可分数据，我们证明了这个充分条件成立。因此，非凸梯度流实际上收敛于一个凸优化问题的最优解。我们给出了数值结果，验证了我们的非凸次梯度下降理论的预测。
We study non-convex subgradient flows for training two-layer ReLU neural networks from a convex geometry and duality perspective. We characterize the implicit bias of unregularized non-convex gradient flow as convex regularization of an equivalent convex model. We then show that the limit points of non-convex subgradient flows can be identified via primal-dual correspondence in this convex optimization problem. Moreover, we derive a sufficient condition on the dual variables which ensures that the stationary points of the non-convex objective are the KKT points of the convex objective, thus proving convergence of non-convex gradient flows to the global optimum. For a class of regular training data distributions such as orthogonal separable data, we show that this sufficient condition holds. Therefore, non-convex gradient flows in fact converge to optimal solutions of a convex optimization problem. We present numerical results verifying the predictions of our theory for non-convex subgradient descent.