Universal approximation power of deep residual neural networks via nonlinear control theory
Universal approximation power of deep residual neural networks via nonlinear control theory
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
P. Tabuada;B. Gharesifard
中科院分区：
文献类型：
--
作者：
P. Tabuada;B. Gharesifard
本文通过几何非线性控制解释了深度残差神经网络的通用逼近能力。受最近在剩余网络和控制系统之间建立联系的工作的启发，我们通过要求激活函数或其一个导数满足二次微分方程，为剩余网络提供了具有通用逼近能力的一般充分条件。实际中使用的许多激活函数满足这个假设，确切地或近似地，我们证明了这个属性足以使每层有$n+1$$个神经元的足够深的神经网络在紧集上和关于上确界范数任意地逼近从$\mathbb{R}^n$到$\mathbb{R}^n$的任何连续函数。我们进一步证明了这一结果适用于非常简单的架构，其中权重只需要假设两个值。第一个关键的技术贡献包括有关的通用近似问题的控制系统的合奏对应的剩余网络的可控性，并利用经典的李代数技术来表征可控性。第二个技术贡献是确定单调性之间的桥梁有限集合的可控性和一致逼近紧集。
In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property to be sufficient for an adequately deep neural network with $n+1$ neurons per layer to approximate arbitrarily well, on a compact set and with respect to the supremum norm, any continuous function from $\mathbb{R}^n$ to $\mathbb{R}^n$. We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control systems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is to identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.