How do infinite width bounded norm networks look in function space?
How do infinite width bounded norm networks look in function space?
复制标题
DOI：
--
复制
发表时间：
2019-02
期刊：
影响因子：
--
通讯作者：
Pedro H. P. Savarese;Itay Evron;Daniel Soudry;N. Srebro
中科院分区：
文献类型：
--
作者：
Pedro H. P. Savarese;Itay Evron;Daniel Soudry;N. Srebro
我们考虑这样一个问题：具有无限多个单元(无限宽度)的RELU网络可以捕获哪些函数，但是整个网络的欧几里德范数(系统中所有权重的平方和，除了每个单元的非正则偏倚项外)是有界的；或者等价地，逼近给定函数所需的最小范数是多少。对于函数$f：\mathbb R\right tarrow\mathbb R$和单隐层，我们证明了表示$f$的最小网络范数是$\max(\int|f‘’(X)|dx，|f‘(-\inty)+f’(+\inty)|)$，因此样本的最小范数是用线性样条插值法给出的。
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function. For functions $f : \mathbb R \rightarrow \mathbb R$ and a single hidden layer, we show that the minimal network norm for representing $f$ is $\max(\int |f''(x)| dx, |f'(-\infty) + f'(+\infty)|)$, and hence the minimal norm fit for a sample is given by a linear spline interpolation.