Multigrid Neural Memory
Multigrid Neural Memory
复制标题
DOI：
--
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
T. Huynh;M. Maire;Matthew R. Walter
中科院分区：
文献类型：
--
作者：
T. Huynh;M. Maire;Matthew R. Walter
我们介绍了一种新的方法来赋予神经网络紧急，长期，大规模的记忆。与通过复杂制作的控制器和手工设计的注意力机制将神经网络连接到外部记忆库的策略不同，我们的记忆是内部的，分布式的，与计算共存，并且隐式寻址，同时比以前的努力要简单得多。架构网络的多重网格结构和连接，而分布在整个拓扑结构的计算存储单元，我们观察到连贯的内存子系统的出现。我们的分层空间组织，卷积参数化，允许大容量存储器的有效实例化，而多重网格拓扑提供了短的内部路由路径，允许卷积网络有效地近似完全连接网络的行为。这种网络具有内在注意力的内隐能力;随着记忆的增强，它们学会以动态的数据依赖方式读取和写入特定的记忆位置。我们在探索和映射任务中展示了这些能力，我们的网络能够自组织并保留数千个时间步的轨迹的长期记忆。在与任何空间几何概念解耦的任务上：排序，联想回忆和问题回答，我们的设计功能作为一个真正的通用记忆，并产生出色的结果。
We introduce a novel approach to endowing neural networks with emergent, long-term, large-scale memory. Distinct from strategies that connect neural networks to external memory banks via intricately crafted controllers and hand-designed attentional mechanisms, our memory is internal, distributed, co-located alongside computation, and implicitly addressed, while being drastically simpler than prior efforts. Architecting networks with multigrid structure and connectivity, while distributing memory cells alongside computation throughout this topology, we observe the emergence of coherent memory subsystems. Our hierarchical spatial organization, parameterized convolutionally, permits efficient instantiation of large-capacity memories, while multigrid topology provides short internal routing pathways, allowing convolutional networks to efficiently approximate the behavior of fully connected networks. Such networks have an implicit capacity for internal attention; augmented with memory, they learn to read and write specific memory locations in a dynamic data-dependent manner. We demonstrate these capabilities on exploration and mapping tasks, where our network is able to self-organize and retain long-term memory for trajectories of thousands of time steps. On tasks decoupled from any notion of spatial geometry: sorting, associative recall, and question answering, our design functions as a truly generic memory and yields excellent results.