The Local Elasticity of Neural Networks
The Local Elasticity of Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Hangfeng He;Weijie J. Su
中科院分区：
文献类型：
--
作者：
Hangfeng He;Weijie J. Su
本文提出了神经网络中的一种现象，我们称之为局部弹性。粗略地说，如果分类器在特征向量$\bx '$上的预测是\texit {not}显著扰动的，则分类器通过随机梯度下降在一定意义上与$\bx'$\texit {不相似}的（标记）特征向量$\bx$上更新后，分类器被称为局部弹性。通过对现实生活和合成数据集的广泛模拟，这种现象在具有非线性激活函数的神经网络中持续存在，而在线性分类器中没有观察到这种现象。此外，我们提供了一个几何解释的局部弹性使用的神经切线内核\citep{jacot 2018 neural}。在局部弹性的基础上，我们获得了特征向量之间的成对相似性度量，它可以与$K$-means一起用于聚类。聚类算法在MNIST和CIFAR-10数据集上的有效性反过来又证实了神经网络对现实数据的局部弹性假设。最后，我们讨论了局部弹性的一些含义，以阐明深度神经网络的几个有趣方面。
This paper presents a phenomenon in neural networks that we refer to as \textit{local elasticity}. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector $\bx'$ is \textit{not} significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector $\bx$ that is \textit{dissimilar} to $\bx'$ in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel \citep{jacot2018neural}. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with $K$-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.