RELU DEEP NEURAL NETWORKS AND LINEAR FINITE ELEMENTS
RELU DEEP NEURAL NETWORKS AND LINEAR FINITE ELEMENTS
复制标题
DOI：
10.4208/jcm.1901-m2018-0160
复制
发表时间：
2020-01-01
影响因子：
0.9
通讯作者：
Zheng, Chunyue
中科院分区：
文献类型：
--
作者：
He, Juncai;Li, Lin;Zheng, Chunyue
在本文中，我们研究了以整流线性单元（ReLU）函数作为激活函数的深度神经网络（DNN）与连续分段线性（CPWL）函数之间的关系，特别是来自单纯线性有限元（FEM）的CPWL函数。我们首先考虑FEM的特殊情况。通过探索其节点基函数的DNN表示，我们提出了有限元中CPWL的ReLU DNN表示。我们从理论上建立了当d >= 2时，ReLU DNN中至少需要2个隐藏层来表示R-d的Omega子集中的任何线性有限元函数。因此，对于在科学和工程计算中经常遇到的d = 2，3，最小数量的两个隐藏层对于任何由ReLU DNN表示的CPWL函数都是必要和足够的。然后，我们详细说明了如何用最多具有左垂直log（2）（d + 1）右垂直隐藏层的ReLU DNN来表示RD中的一般CPWL，并且我们还估计了DNN中需要的神经元数量。此外，利用DNN和FEM之间的关系，我们从理论上论证了一类特殊的低位宽DNN模型在应用中仍然有望具有足够的表示能力。最后，作为概念证明，我们给出了使用ReLU DNN解决两点边界问题的一些数值结果，以证明DNN应用于偏微分方程数值求解的潜力。
In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least 2 hidden layers are needed in a ReLU DNN to represent any linear finite element functions in Omega subset of R-d when d >= 2. Consequently, for d = 2, 3 which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in R d can be represented by a ReLU DNN with at most left perpendicular log(2) (d + 1) right perpendicular hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.