Efficient Approximation of Deep ReLU Networks for Functions on Low Dimensional Manifolds
Efficient Approximation of Deep ReLU Networks for Functions on Low Dimensional Manifolds
复制标题
DOI：
--
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Minshuo Chen;Haoming Jiang;Wenjing Liao;T. Zhao
中科院分区：
文献类型：
--
作者：
Minshuo Chen;Haoming Jiang;Wenjing Liao;T. Zhao
深度神经网络由于其数据拟合的灵活性和对未见数据的准确预测而彻底改变了许多现实世界的应用。一系列研究表明，神经网络可以以任意精度逼近某些类别的函数，而网络的大小相对于数据维度呈指数级增长。然而，经验结果表明，中等规模的网络已经产生了吸引人的性能。为了解释这种差距，人们普遍认为许多数据集表现出低维结构，并且可以建模为低维流形附近的样本。在本文中，我们证明神经网络可以有效地逼近低维流形上支持的函数。网络规模的近似误差呈指数级增长，指数取决于数据的内在维度和函数的平滑度。我们的结果表明，利用低维数据结构可以大大提高神经网络函数逼近的效率。我们还实现了一个子网络，将输入数据分配给它们相应的本地邻域，这些邻域可能具有独立的兴趣。
Deep neural networks have revolutionized many real world applications, due to their flexibility in data fitting and accurate predictions for unseen data. A line of research reveals that neural networks can approximate certain classes of functions with an arbitrary accuracy, while the size of the network scales exponentially with respect to the data dimension. Empirical results, however, suggest that networks of moderate size already yield appealing performance. To explain such a gap, a common belief is that many data sets exhibit low dimensional structures, and can be modeled as samples near a low dimensional manifold. In this paper, we prove that neural networks can efficiently approximate functions supported on low dimensional manifolds. The network size scales exponentially in the approximation error, with an exponent depending on the intrinsic dimension of the data and the smoothness of the function. Our result shows that exploiting low dimensional data structures can greatly enhance the efficiency in function approximation by neural networks. We also implement a sub-network that assigns input data to their corresponding local neighborhoods, which may be of independent interest.