Biased ReLU neural networks
Biased ReLU neural networks
复制标题
有偏差的 ReLU 神经网络
DOI：
10.1016/j.neucom.2020.09.050
复制
发表时间：
2021-01-29
期刊：
影响因子：
6
通讯作者：
Xu, Jun
中科院分区：
文献类型：
--
作者：
Liang, XingLong;Xu, Jun
自2012年以来，具有整流线性单元（RELU）的神经网络（NN）已被广泛实施。在本文中，我们描述了一种称为偏置的Relu神经元（BRELU）的激活函数，该函数与Relu相似。基于此激活函数，我们提出了Brelu nn（BRNN）。 BRNN的结构与Relu网络的结构相似。但是，两者之间的区别在于，Brelu为每个输入变量引入了几个偏差。这使BRNN可以将输入空间分为更多的线性区域并提高网络灵活性。要估计的BRNN参数是Brelu神经元的重量矩阵和偏置参数。使用反向传播方法获得权重。此外，我们提出了一种计算Brelu神经元偏置参数的方法。在此方法中，将批准化应用于BRNN，并获得输入变量的方差和平均值。基于这两个参数，估计偏差参数。此外，我们研究了BRNN的灵活性。具体而言，我们研究线性区域的数量，并为最大线性区域数量提供上限。结果表明，对于相同的输入维度，BRNN将输入空间分为比Relu网络更大的线性区域。这在一定程度上解释了为什么BRNN具有出色的近似能力。实验是使用五个数据集进行的，结果验证了所提出方法的有效性。 （c）2020 Elsevier B.V.保留所有权利。
Neural networks (NN) with rectified linear units (ReLU) have been widely implemented since 2012. In this paper, we describe an activation function called the biased ReLU neuron (BReLU), which is similar to the ReLU. Based on this activation function, we propose the BReLU NN (BRNN). The structure of the BRNN is similar to that of the ReLU network. However, the difference between the two is that the BReLU introduces several biases for each input variable. This allows the BRNN to divide the input space into a greater number of linear regions and improve network flexibility. The BRNN parameters to be estimated are the weight matrices and the bias parameters of the BReLU neurons. The weights are obtained using the backpropagation method. Moreover, we propose a method to compute the bias parameters of the BReLU neurons. In this method, batch normalization is applied to the BRNN, and the variance and mean of the input variables are obtained. Based on these two parameters, the bias parameters are estimated. In addition, we investigate the flexibility of the BRNN. Specifically, we study the number of linear regions and provide the upper bound for the maximum number of linear regions. The results indicate that for the same input dimension, the BRNN divides the input space into a greater number of linear regions than the ReLU network. This explains to a certain extent why the BRNN has the superior approximation ability. Experiments are carried out using five datasets, and the results verify the effectiveness of the proposed method. (c) 2020 Elsevier B.V. All rights reserved.