How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
复制标题
DOI：
--
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Zixiang Chen;Yuan Cao;Difan Zou;Quanquan Gu
中科院分区：
文献类型：
--
作者：
Zixiang Chen;Yuan Cao;Difan Zou;Quanquan Gu
最近关于深度学习的一系列研究集中在极度过度参数化的设置上，并表明当网络宽度大于训练样本大小 $n$ 的高次多项式和目标精度的倒数 $\epsilon^{-1}$ 时，通过（随机）梯度下降学习的深度神经网络享有良好的优化和泛化保证。最近，研究表明，在训练数据的一定裕度假设下，多对数宽度条件足以使两层 ReLU 网络收敛和泛化（Ji 和 Telgarsky，2019）。然而，多少过度参数化足以保证深度神经网络的优化和泛化仍然是一个悬而未决的问题。在这项工作中，我们为深度 ReLU 网络建立了敏锐的优化和泛化保证。在之前工作中做出的各种假设下，我们的优化和泛化保证在 $n$ 和 $\epsilon^{-1}$ 中保持网络宽度多对数。我们的结果推动了过度参数化深度神经网络的研究走向更实际的环境。
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target accuracy $\epsilon^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees. Very recently, it is shown that under certain margin assumption on the training data, a polylogarithmic width condition suffices for two-layer ReLU networks to converge and generalize (Ji and Telgarsky, 2019). However, how much over-parameterization is sufficient to guarantee optimization and generalization for deep neural networks still remains an open question. In this work, we establish sharp optimization and generalization guarantees for deep ReLU networks. Under various assumptions made in previous work, our optimization and generalization guarantees hold with network width polylogarithmic in $n$ and $\epsilon^{-1}$. Our results push the study of over-parameterized deep neural networks towards more practical settings.