Efficient Decoding of Compositional Structure in Holistic Representations
Efficient Decoding of Compositional Structure in Holistic Representations
复制标题
DOI：
10.1162/neco_a_01590
复制
发表时间：
2023-07-01
影响因子：
2.9
通讯作者：
Sommer，Friedrich T.
中科院分区：
文献类型：
--
作者：
Kleyko，Denis;Bybee，Connor;Sommer，Friedrich T.
我们调查的任务组成的分布式表示形成的多维计算/矢量符号架构检索信息，并提出新的技术，实现新的信息速率界限。首先，我们提供了一个概述的解码技术，可以用来接近检索任务。这些技术分为四组。然后，我们评估所考虑的技术在几个设置，涉及，例如，包括外部噪声和存储元件的精度降低。特别是，我们发现，从稀疏编码和压缩感知文献（很少用于多维计算/矢量符号架构）的解码技术也非常适合解码信息的组成分布式表示。将这些解码技术与来自通信的干扰消除思想相结合改进了先前报告的界限（Hersche等人，对于较小的码本，分布式表示的信息速率为每维1.20至1.40比特，对于较大的码本，分布式表示的信息速率为每维0.60至1.26比特。
We investigate the task of retrieving information from compositional distributed representations formed by hyperdimensional computing/vector symbolic architectures and present novel techniques that achieve new information rate bounds. First, we provide an overview of the decoding techniques that can be used to approach the retrieval task. The techniques are categorized into four groups. We then evaluate the considered techniques in several settings that involve, for example, inclusion of external noise and storage elements with reduced precision. In particular, we find that the decoding techniques from the sparse coding and compressed sensing literature (rarely used for hyperdimensional computing/vector symbolic architectures) are also well suited for decoding information from the compositional distributed representations. Combining these decoding techniques with interference cancellation ideas from communications improves previously reported bounds (Hersche et al., ) of the information rate of the distributed representations from 1.20 to 1.40 bits per dimension for smaller codebooks and from 0.60 to 1.26 bits per dimension for larger codebooks.