A representer theorem for deep neural networks
A representer theorem for deep neural networks
复制标题
DOI：
--
复制
发表时间：
2018-02
期刊：
影响因子：
--
通讯作者：
M. Unser
中科院分区：
文献类型：
--
作者：
M. Unser
我们建议通过在成本函数中添加相应的功能正则化来优化深神经网络的激活功能。我们证明使用二阶总变量标准是合理的。这使我们能够为深度神经网络提供一般代表定理，该定理与花键和稀疏性直接连接。具体而言，我们表明，可以使用具有自适应结的非均匀线性细条的激活函数来实现最佳网络配置。最重要的是，每个神经元的动作是由在训练过程中优化参数（包括结的参数（包括结数）的样条的。该方案产生了与现有的Deep-Relu和Maxout体系结构兼容的计算结构。它还提出了新颖的优化挑战，同时用$ \ ell_1 $最小化和刺激性促进技术明确地链接。
We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connection with splines and sparsity. Specifically, we show that the optimal network configuration can be achieved with activation functions that are nonuniform linear splines with adaptive knots. The bottom line is that the action of each neuron is encoded by a spline whose parameters (including the number of knots) are optimized during the training procedure. The scheme results in a computational structure that is compatible with the existing deep-ReLU and MaxOut architectures. It also suggests novel optimization challenges, while making the link with $\ell_1$ minimization and sparsity-promoting techniques explicit.