Provable Memorization via Deep Neural Networks using Sub-linear Parameters
Provable Memorization via Deep Neural Networks using Sub-linear Parameters
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Sejun Park;Jaeho Lee;Chulhee Yun;Jinwoo Shin
中科院分区：
文献类型：
--
作者：
Sejun Park;Jaeho Lee;Chulhee Yun;Jinwoo Shin
众所周知，$\Theta（N）$参数足以让神经网络记住任意$N$个输入-标签对。通过利用深度，我们表明，$\Theta（N^{2/3}）$参数足以记住$N$对，在一个温和的条件下，对输入点的分离。特别是，更深的网络（即使宽度为3美元）被证明比浅网络记住更多的对，这也与最近关于深度对函数近似的好处的工作一致。我们还提供了实证结果，支持我们的理论研究结果。
It is known that $\Theta(N)$ parameters are sufficient for neural networks to memorize arbitrary $N$ input-label pairs. By exploiting depth, we show that $\Theta(N^{2/3})$ parameters suffice to memorize $N$ pairs, under a mild condition on the separation of input points. In particular, deeper networks (even with width $3$) are shown to memorize more pairs than shallow networks, which also agrees with the recent line of works on the benefits of depth for function approximation. We also provide empirical results that support our theoretical findings.