Nonclosedness of sets of neural networks in Sobolev spaces
Nonclosedness of sets of neural networks in Sobolev spaces
复制标题
DOI：
10.1016/j.neunet.2021.01.007
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Scott Mahan;E. King;A. Cloninger
中科院分区：
文献类型：
--
作者：
Scott Mahan;E. King;A. Cloninger
我们研究了Sobolev空间中固定结构的神经网络集的封闭性。对于一个精确的m次可微激活函数ρ，我们构造了一个神经网络序列（Φ n） n∈n，其实现以order-(m−1)Sobolev范数收敛于一个不能由神经网络精确实现的函数。因此，对于p∈[1，∞]，所实现的神经网络集合在order-(m−1)Sobolev空间W m−1，p中不闭合。我们进一步证明了这些集合在W m， p中是不闭合的在ρ的m阶导数的稍强的条件下。对于一个真实的解析激活函数，我们证明了对于任意k∈n，所实现的神经网络集在W k， p中是不闭合的。这种非闭合性允许具有无界参数增长的非网络目标函数的逼近。我们通过表明特定的神经网络序列可以近似激活函数的导数来部分表征大多数激活函数的参数增长速度，其权重的增加与lp近似误差成反比。最后，我们给出了实验结果，表明网络能够通过训练接近非网络目标函数，并增加参数。
We examine the closedness of sets of realized neural networks of a fixed architecture in Sobolev spaces. For an exactly m-times differentiable activation function ρ, we construct a sequence of neural networks (Φ n) n∈ N whose realizations converge in order-(m− 1) Sobolev norm to a function that cannot be realized exactly by a neural network. Thus, sets of realized neural networks are not closed in order-(m− 1) Sobolev spaces W m− 1, p for p∈[1,∞). We further show that these sets are not closed in W m, p under slightly stronger conditions on the m th derivative of ρ. For a real analytic activation function, we show that sets of realized neural networks are not closed in W k, p for any k∈ N. The nonclosedness allows for approximation of non-network target functions with unbounded parameter growth. We partially characterize the rate of parameter growth for most activation functions by showing that a specific sequence of realized neural networks can approximate the activation function’s derivative with weights increasing inversely proportional to the L p approximation error. Finally, we present experimental results showing that networks are capable of closely approximating non-network target functions with increasing parameters via training.