Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Alex Lamb;Anirudh Goyal;A. Slowik;M. Mozer;Philippe Beaudoin;Y. Bengio
中科院分区：
文献类型：
--
作者：
Alex Lamb;Anirudh Goyal;A. Slowik;M. Mozer;Philippe Beaudoin;Y. Bengio
前馈神经网络由一系列层组成，其中每一层都对前一层的信息进行一些处理。这种方法的缺点是，每个层（或模块，因为多个模块可以并行操作）的任务是处理整个隐藏状态，而不是与该模块最相关的状态的特定部分。只对少量输入变量进行操作的方法是大多数编程语言的重要组成部分，它们允许改进的模块化和代码重用性。我们提出的方法，神经功能模块（NFM），旨在将相同的结构能力引入深度学习。结合自上而下和自下而上反馈的前馈网络的大部分工作仅限于分类问题。我们工作的主要贡献是在一个灵活的算法中结合联合收割机注意力，稀疏性，自上而下和自下而上的反馈，正如我们所展示的那样，在强化学习的背景下，改进了标准分类，域外泛化，生成建模和学习表示的结果。
Feed-forward neural networks consist of a sequence of layers, in which each layer performs some processing on the information from the previous layer. A downside to this approach is that each layer (or module, as multiple modules can operate in parallel) is tasked with processing the entire hidden state, rather than a particular part of the state which is most relevant for that module. Methods which only operate on a small number of input variables are an essential part of most programming languages, and they allow for improved modularity and code re-usability. Our proposed method, Neural Function Modules (NFM), aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm which, as we show, improves the results in standard classification, out-of-domain generalization, generative modeling, and learning representations in the context of reinforcement learning.