Coordination Among Neural Modules Through a Shared Global Workspace
Coordination Among Neural Modules Through a Shared Global Workspace
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Anirudh Goyal;Aniket Didolkar;Alex Lamb;Kartikeya Badola;Nan Rosemary Ke;Nasim Rahaman;Jonathan Binas;C. Blundell;M. Mozer;Y. Bengio
中科院分区：
文献类型：
--
作者：
Anirudh Goyal;Aniket Didolkar;Alex Lamb;Kartikeya Badola;Nan Rosemary Ke;Nasim Rahaman;Jonathan Binas;C. Blundell;M. Mozer;Y. Bengio
深度学习已经看到了一种趋势，即从用单一的隐藏状态来表示例子，转向结构丰富的状态。例如，Transformers按位置分段，而以对象为中心的体系结构将图像分解为实体。在所有这些体系结构中，不同元素之间的交互是通过成对交互来建模的：变形金刚利用自我注意来整合来自其他位置的信息；以对象为中心的体系结构利用图神经网络来建模实体之间的交互。然而，成对交互可能无法实现全局协调或可用于下游任务的连贯、集成的表示。在认知科学中，已经提出了一种全球工作空间体系结构，其中功能专门的组件通过公共的、带宽有限的通信通道来共享信息。我们探索在深度学习的背景下使用这样的通信渠道来对复杂环境的结构进行建模。拟议的方法包括一个共享工作空间，不同专家模块之间通过该空间进行通信，但由于通信带宽的限制，专家模块必须竞争访问权限。我们表明，能力限制有一个合理的基础，即(1)它们鼓励专业化和组合性，(2)它们促进其他独立专家的同步。
Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.