On the Impact of the Activation Function on Deep Neural Networks Training
On the Impact of the Activation Function on Deep Neural Networks Training
复制标题
DOI：
--
复制
发表时间：
2019-02
期刊：
影响因子：
--
通讯作者：
Soufiane Hayou;A. Doucet;J. Rousseau
中科院分区：
文献类型：
--
作者：
Soufiane Hayou;A. Doucet;J. Rousseau
深度神经网络的权值初始化和激活函数对训练过程的性能有着至关重要的影响。不适当的选择可能会导致前向传播过程中输入信息的丢失和后向传播过程中梯度的指数消失/爆炸。了解未经训练的随机网络的理论属性是确定哪些深度网络可能被成功训练的关键，Samuel等人(2017)最近证明，对于深度前馈神经网络，只有特定选择的被称为“混沌边缘”的超参数才能产生良好的性能。虽然Samuel等人(2017)的工作讨论了可训练性问题，但我们在这里重点关注训练加速和整体性能。我们对混沌边缘进行了全面的理论分析，证明了我们确实可以通过调整初始化参数和激活函数来加快训练速度，提高性能。
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Samuel et al (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `Edge of Chaos' can lead to good performance. While the work by Samuel et al (2017) discuss trainability issues, we focus here on training acceleration and overall performance. We give a comprehensive theoretical analysis of the Edge of Chaos and show that we can indeed tune the initialization parameters and the activation function in order to accelerate the training and improve the performance.