ResNet and Batch-normalization Improve Data Separability
ResNet and Batch-normalization Improve Data Separability
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Yasutaka Furusho;K. Ikeda
中科院分区：
文献类型：
--
作者：
Yasutaka Furusho;K. Ikeda
ResNet中的跳跃连接和批量归一化（BN）使极深神经网络能够以高性能进行训练。然而，其高性能的原因尚不清楚。为了明确这一点，我们研究了跳过连接和BN对通过隐藏层的类相关信号传播的影响，因为最后一个隐藏层的特征向量的类间距离与类内距离的大比例会导致高性能。我们的研究结果表明，类间距离和类内距离的变化不同，通过层：随机初始化的权重的深层多层感知器降低类间距离的比例，类内距离和跳跃连接和BN放松这种退化。此外，我们的分析表明，跳跃连接和BN鼓励训练，以提高这个距离比。这些结果表明，跳跃连接和BN诱导高性能。
The skip-connection and the batch-normalization (BN) in ResNet enable an extreme deep neural network to be trained with high performance. However, the reasons for its high performance are still unclear. To clear that, we study the effects of the skip-connection and the BN on the class-related signal propagation through hidden layers because a large ratio of the between-class distance to the within-class distance of feature vectors at the last hidden layer induces high performance. Our result shows that the between-class distance and the within-class distance change differently through layers: the deep multilayer perceptron with randomly initialized weights degrades the ratio of the between-class distance to the within-class distance and the skip-connection and the BN relax this degradation. Moreover, our analysis implies that the skip-connection and the BN encourage training to improve this distance ratio. These results imply that the skip-connection and the BN induce high performance.