Power Law in Sparsified Deep Neural Networks
Power Law in Sparsified Deep Neural Networks
复制标题
稀疏深度神经网络中的幂律
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
J. Kwok
中科院分区：
文献类型：
--
作者：
Lu Hou;J. Kwok
在许多生物神经网络的度分布中都观察到了幂定律。稀疏深度神经网络从数据中学习一种经济的表示法，在许多方面类似于生物神经网络。在本文中，我们研究这些人工网络是否也表现出幂定律的性质。在两种流行的深度学习模型，即多层感知器和卷积神经网络上的实验结果是肯定的。权力法则也自然地与优惠依附有关。为了研究深度网络在连续学习中的动力学特性，我们提出了一个内部偏好依恋模型来解释网络拓扑的演化过程。实验结果表明，随着新任务的到来，新的连接遵循这种优先依恋过程。
The power law has been observed in the degree distributions of many biological neural networks. Sparse deep neural networks, which learn an economical representation from the data, resemble biological neural networks in many ways. In this paper, we study if these artificial networks also exhibit properties of the power law. Experimental results on two popular deep learning models, namely, multilayer perceptrons and convolutional neural networks, are affirmative. The power law is also naturally related to preferential attachment. To study the dynamical properties of deep networks in continual learning, we propose an internal preferential attachment model to explain how the network topology evolves. Experimental results show that with the arrival of a new task, the new connections made follow this preferential attachment process.
DOI：
10.1073/pnas.200327197
发表时间：
2000-10-10
影响因子：
11.1
作者：
Amaral, LAN;Scala, A;Stanley, HE
通讯作者：
Stanley, HE
影响因子：
8.6
作者：
Eguíluz, VM;Chialvo, DR;Apkarian, AV
通讯作者：
Apkarian, AV