Variable Binding for Sparse Distributed Representations: Theory and Applications
Variable Binding for Sparse Distributed Representations: Theory and Applications
复制标题
DOI：
10.1109/tnnls.2021.3105949
复制
发表时间：
2021-09-02
影响因子：
10.4
通讯作者：
Sommer, Friedrich T.
中科院分区：
文献类型：
--
作者：
Frady, Edward Paxon;Kleyko, Denis;Sommer, Friedrich T.
变量绑定是符号推理和认知的基石。但是如何在联结主义模型中实现绑定，已经困扰了神经科学家、认知心理学家和神经网络研究人员几十年。自然包括绑定操作的一种类型的连接模型是向量符号架构（VSAs）。与变量绑定的其他提议相比，VSA中的绑定操作是维保持的，这使得能够表示复杂的分层数据结构，例如树，同时避免维度的组合扩展。经典的VSA通过密集的随机向量对符号进行编码，其中信息分布在整个神经元群体中。相比之下，在大脑中，特征被编码得更局部，由单个神经元或小神经元组的活动编码，通常形成神经激活的稀疏向量。继Laiho等人（2015）之后，我们探索了稀疏分布表示的特殊情况下的符号推理。使用压缩传感技术，我们首先证明经典VSA中的变量绑定在数学上等效于稀疏特征向量之间的张量积绑定，这是另一种众所周知的增加维度的绑定操作。这一理论结果促使我们研究两个维保持绑定方法，包括减少张量矩阵到一个单一的稀疏向量。一种是对一般稀疏向量采用随机投影的绑定方法，另一种是对具有块结构的稀疏向量，即稀疏块码，采用块局部循环卷积的绑定方法。我们的实验表明，块局部循环卷积绑定具有理想的性能，而基于随机投影的绑定也可以工作，但有损耗。我们在示例应用中证明，具有块局部循环卷积和稀疏块码的VSA达到与经典VSA相似的性能。最后，我们在神经科学和神经网络的背景下讨论我们的结果。
Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the context of neuroscience and neural networks.