Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines
Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines
复制标题
DOI：
10.3390/e19070310
复制
发表时间：
2017-07-01
期刊：
影响因子：
2.7
通讯作者：
Ay, Nihat
中科院分区：
文献类型：
--
作者：
Kanwal, Maxinder S.;Grochow, Joshua A.;Ay, Nihat
在过去的三十年里，许多复杂性的理论度量被提出来帮助理解复杂系统。在这项工作中，我们第一次将这些措施放在一个公平的竞争环境中，探讨它们之间的质的相似性和差异，以及它们的缺点。具体来说，使用具有均匀分布权重的玻尔兹曼机架构（一种完全连接的递归神经网络）作为我们的研究模型，我们在数值上测量复杂性如何随网络动态和网络参数而变化。我们应用这样一个信息理论的复杂性措施的扩展，以了解增量赫布学习Hopfield网络，一个完全经常性的自联想记忆的架构模型。在赫布学习过程中，随着网络试图学习越来越多的模式，总的信息流反映了复杂性的自然上升趋势。
In the past three decades, many theoretical measures of complexity have been proposed to help understand complex systems. In this work, for the first time, we place these measures on a level playing field, to explore the qualitative similarities and differences between them, and their shortcomings. Specifically, using the Boltzmann machine architecture (a fully connected recurrent neural network) with uniformly distributed weights as our model of study, we numerically measure how complexity changes as a function of network dynamics and network parameters. We apply an extension of one such information-theoretic measure of complexity to understand incremental Hebbian learning in Hopfield networks, a fully recurrent architecture model of autoassociative memory. In the course of Hebbian learning, the total information flow reflects a natural upward trend in complexity as the network attempts to learn more and more patterns.