A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines
A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines
复制标题
DOI：
10.1016/j.jcp.2021.110585
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
S. Dong;Zongwei Li
中科院分区：
文献类型：
--
作者：
S. Dong;Zongwei Li
在极端学习机（ELM）中，隐层系数是随机设置和固定的，而神经网络的输出层系数是通过最小二乘法计算的。众所周知，ELM中随机分配的系数会显着影响其性能和准确性。在本文中，我们提出了一种改进的批内禀塑性（modBIP）方法预训练的随机系数的ELM神经网络。该方法与批处理本征塑性（BIP）方法基于相同的原理，即通过增强神经网络中每个节点的信息传输来设计。它在两个突出方面不同于BIP。首先，modBIP在其算法中不涉及激活函数，并且它可以与神经网络中的任何激活函数一起应用。相反，BIP在其构造中使用激活函数的逆，并且要求激活函数是可逆的（或单调的）。modBIP方法可以与经常使用的非单调激活函数（例如高斯，swish，高斯误差线性单元和径向基类型函数）一起工作，BIP可以与这些函数一起分解。其次，modBIP以最小大小的随机间隔生成目标样本，与ELM结合使用时可获得高度准确的计算结果。在数值模拟中，组合ELM/modBIP方法的精度明显高于ELM/BIP方法。用浅层和深层神经网络对偏微分方程的函数逼近和边值/初值问题进行了充分的数值实验。他们表明，组合ELM/modBIP方法产生高度准确的模拟结果，其准确性是不敏感的随机系数初始化的神经网络。这与没有预先训练随机系数的ELM结果形成鲜明对比。
In extreme learning machines (ELM) the hidden-layer coefficients are randomly set and fixed, while the output-layer coefficients of the neural network are computed by a least squares method. The randomly-assigned coefficients in ELM are known to influence its performance and accuracy significantly. In this paper we present a modified batch intrinsic plasticity (modBIP) method for pre-training the random coefficients in the ELM neural networks. The current method is devised based on the same principle as the batch intrinsic plasticity (BIP) method, namely, by enhancing the information transmission in every node of the neural network. It differs from BIP in two prominent aspects. First, modBIP does not involve the activation function in its algorithm, and it can be applied with any activation function in the neural network. In contrast, BIP employs the inverse of the activation function in its construction, and requires the activation function to be invertible (or monotonic). The modBIP method can work with the often-used non-monotonic activation functions (e.g. Gaussian, swish, Gaussian error linear unit, and radial-basis type functions), with which BIP breaks down. Second, modBIP generates target samples on random intervals with a minimum size, which leads to highly accurate computation results when combined with ELM. The combined ELM/modBIP method is markedly more accurate than ELM/BIP in numerical simulations. Ample numerical experiments are presented with shallow and deep neural networks for function approximation and boundary/initial value problems with partial differential equations. They demonstrate that the combined ELM/modBIP method produces highly accurate simulation results, and that its accuracy is insensitive to the random-coefficient initializations in the neural network. This is in sharp contrast with the ELM results without pre-training of the random coefficients.