Deep Network Approximation Characterized by Number of Neurons
Deep Network Approximation Characterized by Number of Neurons
复制标题
DOI：
10.4208/cicp.oa-2020-0149
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Zuowei Shen;Haizhao Yang;Shijun Zhang
中科院分区：
文献类型：
--
作者：
Zuowei Shen;Haizhao Yang;Shijun Zhang
本文定量地表征了深馈神经网络（FNN）的近似能力，即神经元的数量，即网络宽度和深度的乘积。通过构造显示的是，具有宽度$ \ mywidth $和深度$ 9L+12 $的Relu fnns可以近似订单$ \ alpha $的任意持有人使用Lipschitz常数$ \ nu $ [0,1]^ d $带紧密近似率$ 5（8 \ sqrt {d}）^\ alpha \ nu n^{ - 2 \ alpha/d} l^{ - 2 \ alpha/d} $对于任何给定的$ n，l \ in \ n^+$。在其连续性$ \ omega_f（\ cdot）$方面，建设性近似是任意连续函数$ f $的更一般结果的推论。特别是，宽度$ \ mywidth $和深度$ 9L+12 $的近似值的近似率$ 5 ^{ - 2/d}）$。我们还将分析扩展到$ f $的域不规则或本地化为$ \ epsilon $ -Neighborhood a $ d _ {\ MATHCAL {M}} $ - 尺寸平滑歧管$ \ MATHCAL {M MATHCAL {M MATHCAL {M MATHCAL {M MATHCAL {M MATHCAL {M MATHCAL {M MATHCAL {M} \ subseteq [0,1]^d $带有$ d _ {\ mathcal {m}}} \ ll d $。特别是在基本低维域的情况下，我们显示近似率$ 3 \ omega_f \ big（\ tfrac {4 \ epsilon} {1- \ delta} \ sqrt {\ tfrac {d} {d_ \ delta}}}} \ big）+5 \ o mega_f \ big（\ tfrac {16d} {（1- \ delta）\ sqrt {d_ \ delta}} n^{ - 2/d_ \ delta} l^{ - 2/d_2/d_ \ delta } \ big）$ for relu fnns to $ \ epsilon $ -neighborhood在$ \ epsilon $ neighborhood中，其中$ d_ \ delta = \ oo \ big（d _ {\ Mathcal {M}}}} \ tfrac {\ tfrac { delta）} {\ delta^2} \ big）$） $ \ delta \ in（0,1）$。我们的分析提供了一个通用指南，以选择宽度和深度的连续函数，尤其是在平行计算中。
This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons, i.e., the product of the network width and depth. It is shown by construction that ReLU FNNs with width $\mywidth$ and depth $9L+12$ can approximate an arbitrary Holder continuous function of order $\alpha$ with a Lipschitz constant $\nu$ on $[0,1]^d$ with a tight approximation rate $5(8\sqrt{d})^\alpha\nu N^{-2\alpha/d}L^{-2\alpha/d}$ for any given $N,L\in \N^+$. The constructive approximation is a corollary of a more general result for an arbitrary continuous function $f$ in terms of its modulus of continuity $\omega_f(\cdot)$. In particular, the approximation rate of ReLU FNNs with width $\mywidth$ and depth $9L+12$ for a general continuous function $f$ is $5\omega_f(8\sqrt{d} N^{-2/d}L^{-2/d})$. We also extend our analysis to the case when the domain of $f$ is irregular or localized in an $\epsilon$-neighborhood of a $d_{\mathcal{M}}$-dimensional smooth manifold $\mathcal{M}\subseteq [0,1]^d$ with $d_{\mathcal{M}}\ll d$. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate $3\omega_f\big(\tfrac{4\epsilon}{1-\delta}\sqrt{\tfrac{d}{d_\delta}}\big)+5\omega_f\big(\tfrac{16d}{(1-\delta)\sqrt{d_\delta}}N^{-2/d_\delta}L^{-2/d_\delta }\big)$ for ReLU FNNs to approximate $f$ in the $\epsilon$-neighborhood, where $d_\delta=\OO\big(d_{\mathcal{M}}\tfrac{\ln (d/\delta)}{\delta^2}\big)$ for any given $\delta\in(0,1)$. Our analysis provides a general guide for selecting the width and the depth of ReLU FNNs to approximate continuous functions especially in parallel computing.