Deep Neural Network Structures Solving Variational Inequalities
Deep Neural Network Structures Solving Variational Inequalities
复制标题
DOI：
10.1007/s11228-019-00526-z
复制
发表时间：
2020-09-01
影响因子：
1.6
通讯作者：
Pesquet, Jean-Christophe
中科院分区：
文献类型：
--
作者：
Combettes, Patrick L.;Pesquet, Jean-Christophe
受深度神经网络中出现的结构的启发，我们研究了定义在不同空间上的交替邻近算子和仿射算子的非线性复合模型。我们首先证明了在神经网络中使用的广泛的激活算子实际上是接近算子。然后，我们建立了所提出的复合结构的平均值的条件，并研究了它们的渐近性质。结果表明，所得到的过程的极限解决了一个变分不等式，该变分不等式一般不是从极小化问题得到的。该分析依赖于单调算子理论的工具，并揭示了一类具有迄今难以捉摸的渐近性质的神经网络结构。
Motivated by structures that appear in deep neural networks, we investigate nonlinear composite models alternating proximity and affine operators defined on different spaces. We first show that a wide range of activation operators used in neural networks are actually proximity operators. We then establish conditions for the averagedness of the proposed composite constructs and investigate their asymptotic properties. It is shown that the limit of the resulting process solves a variational inequality which, in general, does not derive from a minimization problem. The analysis relies on tools from monotone operator theory and sheds some light on a class of neural networks structures with so far elusive asymptotic properties.