Adaptively Customizing Activation Functions for Various Layers
Adaptively Customizing Activation Functions for Various Layers
复制标题
自适应地定制各个层的激活函数
DOI：
10.1109/tnnls.2021.3133263
复制
发表时间：
2021-12
影响因子：
10.4
通讯作者：
Zhou Qianwei
中科院分区：
文献类型：
--
作者：
Hu Haigen;Liu Aizhu;Guan Qiu;Qian Hanwang;Li Xiaoxin;Chen Shengyong;Zhou Qianwei
为了增强神经网络的非线性并提高其在输入和响应变量之间的映射能力，激活函数在对数据中更复杂的关系和模式进行建模方面起着至关重要的作用。在这项工作中，一种新颖的……（“met”可能是未完整的单词，如“method”方法等）
To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this work, a novel methodology is proposed to adaptively customize activation functions only by adding very few parameters to the traditional activation functions such as Sigmoid, Tanh, and rectified linear unit (ReLU). To verify the effectiveness of the proposed methodology, some theoretical and experimental analysis on accelerating the convergence and improving the performance is presented, and a series of experiments are conducted based on various network models (such as AlexNet, VggNet, GoogLeNet, ResNet and DenseNet), and various datasets (such as CIFAR10, CIFAR100, miniImageNet, PASCAL VOC, and COCO). To further verify the validity and suitability in various optimization strategies and usage scenarios, some comparison experiments are also implemented among different optimization strategies (such as SGD, Momentum, AdaGrad, AdaDelta, and ADAM) and different recognition tasks such as classification and detection. The results show that the proposed methodology is very simple but with significant performance in convergence speed, precision, and generalization, and it can surpass other popular methods such as ReLU and adaptive functions such as Swish in almost all experiments in terms of overall performance.
登录
查看更多内容
DOI：
--
发表时间：
1946
期刊：
--
影响因子：
--
作者：
H. Cramér
通讯作者：
H. Cramér
DOI：
--
发表时间：
2006
期刊：
--
影响因子：
--
作者：
Jianguo Zhang
通讯作者：
Jianguo Zhang
用于视觉跟踪的区域深度学习模型
影响因子：
6
作者：
Wu, Guoxing;Lu, Wenjie;Liu, Jiayin
通讯作者：
Liu, Jiayin
影响因子：
6
作者：
Yunlan Tan;Pengjie Tang;Yimin Zhou;Wenlang Luo;Yongping Kang;Guangyao Li
通讯作者：
Yunlan Tan;Pengjie Tang;Yimin Zhou;Wenlang Luo;Yongping Kang;Guangyao Li
DOI：
--
发表时间：
2006
期刊：
--
影响因子：
--
作者：
M. Everingham;Andrew Zisserman;Christopher K. I. Williams;L. V. Gool
通讯作者：
M. Everingham;Andrew Zisserman;Christopher K. I. Williams;L. V. Gool