Singular Values for ReLU Layers
Singular Values for ReLU Layers
复制标题
DOI：
10.1109/tnnls.2019.2945113
复制
发表时间：
2020-09-01
影响因子：
10.4
通讯作者：
Maass, Peter
中科院分区：
文献类型：
--
作者：
Dittmer, Soeren;King, Emily J.;Maass, Peter
尽管它们在神经网络中普遍存在，但我们仍然缺乏对纠错线性单元(RELU)层的全面的理论表征。本文旨在通过研究激活函数RELU如何与层的线性分量相互作用，以及这种相互作用在神经网络成功实现其预期任务中所起的作用，来加深我们对RELU层的理解。为此，我们引入了两个新的工具：算子的重现奇异值和算子的高斯平均宽度。一方面，通过给出这两个概念的理论依据、结果和解释，另一方面，通过将REU奇异值和高斯平均宽度应用于训练的神经网络的数值实验和结果，我们希望给出REU层的一个全面的、以奇异值为中心的观点。我们发现，RELU奇异值和高斯平均宽度不仅能够提供理论上的见解，而且还提供了似乎有希望用于实际应用的度量。特别是，这些措施可用于在数据遍历网络时区分正确和错误分类的数据。根据我们的发现，我们最后介绍了两种工具：双层和调和剪枝。
Despite their prevalence in neural networks, we still lack a thorough theoretical characterization of rectified linear unit (ReLU) layers. This article aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success of the neural network in achieving its intended task. To this end, we introduce two new tools: ReLU singular values of operators and the Gaussian mean width of operators. By presenting, on the one hand, theoretical justifications, results, and interpretations of these two concepts and, on the other hand, numerical experiments and results of the ReLU singular values and the Gaussian mean width being applied to trained neural networks, we hope to give a comprehensive, singular-value-centric view of ReLU layers. We find that ReLU singular values and the Gaussian mean width do not only enable theoretical insights but also provide one with metrics that seem promising for practical applications. In particular, these measures can be used to distinguish correctly and incorrectly classified data as it traverses the network. We conclude by introducing two tools based on our findings: double layers and harmonic pruning.