A comparison of deep networks with ReLU activation function and linear spline-type methods
A comparison of deep networks with ReLU activation function and linear spline-type methods
复制标题
DOI：
10.1016/j.neunet.2018.11.005
复制
发表时间：
2019-02-01
期刊：
影响因子：
7.8
通讯作者：
Schmidt-Hieber, Johannes
中科院分区：
文献类型：
--
作者：
Eckle, Konstantin;Schmidt-Hieber, Johannes
深度神经网络（DNN）比浅层网络生成更丰富的函数空间。由于浅网络诱导的函数空间有几个近似理论的缺点，这并不一定解释了深网络的成功。在这篇文章中，我们采取了另一种方法，将带有ReLU激活函数的DNN的表达能力与线性样条方法进行比较。我们表明，MARS（多元自适应回归样条）是不适当的学习DNN的意义上，对于任何给定的函数，可以表示为一个函数的MARS与M参数存在一个多层神经网络与O（M log（M/M））参数，逼近这个函数的超范数误差ε。我们展示了一个类似的结果膨胀相对于Faber-Schauder系统。在此基础上，我们推导出风险比较不等式，该不等式通过基于样条的方法的统计风险来约束拟合神经网络的统计风险。这表明深度网络的性能比所考虑的样条方法更好或仅略差。给出了函数逼近的构造性证明。(C)2018作者（S）由爱思唯尔有限公司发布
Deep neural networks (DNNs) generate much richer function spaces than shallow networks. Since the function spaces induced by shallow networks have several approximation theoretic drawbacks, this explains, however, not necessarily the success of deep networks. In this article we take another route by comparing the expressive power of DNNs with ReLU activation function to linear spline methods. We show that MARS (multivariate adaptive regression splines) is improper learnable by DNNs in the sense that for any given function that can be expressed as a function in MARS with M parameters there exists a multilayer neural network with O(M log(M/epsilon)) parameters that approximates this function up to sup-norm error epsilon. We show a similar result for expansions with respect to the Faber-Schauder system. Based on this, we derive risk comparison inequalities that bound the statistical risk of fitting a neural network by the statistical risk of spline-based methods. This shows that deep networks perform better or only slightly worse than the considered spline methods. We provide a constructive proof for the function approximations. (C) 2018 The Author(s). Published by Elsevier Ltd.