Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality
Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality
复制标题
DOI：
--
复制
发表时间：
2019-07
期刊：
影响因子：
--
通讯作者：
Ryumei Nakada;M. Imaizumi
中科院分区：
文献类型：
--
作者：
Ryumei Nakada;M. Imaizumi
在这项研究中，我们证明了协变量的内在低维是决定深度神经网络（dnn）性能的主要因素。dnn通常提供出色的经验性能。因此，许多研究都在积极研究dnn的理论特性，以了解其潜在的机制。特别是，dnn在高维数据方面的行为是最关键的问题之一。然而，尽管高维数据实际上具有较低的内在维数，但从协变量的角度对这一问题的研究还不够充分。在本研究中，我们推导了具有本质低维协变量的深度神经网络的近似误差和泛化误差的界限。我们应用闵可夫斯基维数的概念，提出了一种新的证明方法。因此，我们表明dnn的误差收敛率不依赖于数据的名义高维数，而是依赖于其较低的固有维数。进一步证明了该速率在极大极小意义上是最优的。我们通过显示dnn可以处理比其他自适应估计器更广泛的内禀低维数据来识别dnn的优势。最后，通过数值模拟对理论结果进行了验证。
In this study, we prove that an intrinsic low dimensionality of covariates is the main factor that determines the performance of deep neural networks (DNNs). DNNs generally provide outstanding empirical performance. Hence, numerous studies have actively investigated the theoretical properties of DNNs to understand their underlying mechanisms. In particular, the behavior of DNNs in terms of high-dimensional data is one of the most critical questions. However, this issue has not been sufficiently investigated from the aspect of covariates, although high-dimensional data have practically low intrinsic dimensionality. In this study, we derive bounds for an approximation error and a generalization error regarding DNNs with intrinsically low dimensional covariates. We apply the notion of the Minkowski dimension and develop a novel proof technique. Consequently, we show that convergence rates of the errors by DNNs do not depend on the nominal high dimensionality of data, but on its lower intrinsic dimension. We further prove that the rate is optimal in the minimax sense. We identify an advantage of DNNs by showing that DNNs can handle a broader class of intrinsic low dimensional data than other adaptive estimators. Finally, we conduct a numerical simulation to validate the theoretical results.