Learning to Selectively Update State Neurons in Recurrent Networks
Learning to Selectively Update State Neurons in Recurrent Networks
复制标题
DOI：
10.1145/3340531.3412018
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Thomas Hartvigsen;Cansu Sen;Elke A. Rundensteiner
中科院分区：
文献类型：
--
作者：
Thomas Hartvigsen;Cansu Sen;Elke A. Rundensteiner
循环神经网络 (RNN) 是最先进的顺序学习方法。然而，无论输入数据如何，标准 RNN 在每个时间步都使用相同的计算量来生成其隐藏状态。最近的工作已经开始通过在每一步中强加先验确定的模式来更新状态来解决这种严格的假设。这些方法可以帮助我们深入了解 RNN 的动态，并可能加快推理速度。然而，当前更新策略的预定性质限制了它们的应用。为了克服这个问题，我们设计了第一个完全学习的方法 SA-RNN，它通过在单个隐藏状态神经元的细粒度上预测离散更新模式来增强任何 RNN。这是通过输入数据驱动的更新似然分布的参数化来实现的。与相关方法不同，我们的方法不对更新模式的结构强加任何假设。更好的是，我们的方法在线调整其更新模式，允许根据输入有条件地更新不同的维度。为了了解要更新哪些维度，该模型解决了多目标优化问题，在统一控制的基础上最大化任务性能，同时最小化更新次数。使用涵盖三个连续学习设置的五个公开数据集，我们证明了与最先进的替代方案相比，我们的方法始终以更少的更新实现了更高的准确性。我们还展示了学习稀疏更新大隐藏状态相对于密集更新小隐藏状态的好处。作为一个额外的好处，我们的方法可以直接应用于包含 RNN 架构的各种模型。
Recurrent Neural Networks (RNNs) are the state-of-the-art approach to sequential learning. However, standard RNNs use the same amount of computation to generate their hidden states at each timestep, regardless of the input data. Recent works have begun to tackle this rigid assumption by imposing a priori-determined patterns for updating the states at each step. These approaches could lend insights into the dynamics of RNNs and possibly speed up inference. However, the pre-determined nature of the current update strategies limits their application. To overcome this, we instead design the first fully-learned approach, SA-RNN, that augments any RNN by predicting discrete update patterns at the fine granularity of individual hidden state neurons. This is achieved through the parameterization of a distribution of update-likelihoods driven by the input data. Unlike related methods, our approach imposes no assumptions on the structure of the update patterns. Better yet, our method adapts its update patterns online, allowing different dimensions to be updated conditionally based on the input. To learn which dimensions to update, the model solves a multi-objective optimization problem, maximizing task performance while minimizing the number of updates based on a unified control. Using five publicly-available datasets spanning three sequential learning settings, we demonstrate that our method consistently achieves higher accuracy with fewer updates compared to state-of-the-art alternatives. We also show the benefits of learning to sparsely-update a large hidden state as opposed to densely-update a small hidden state. As an added benefit, our method can be directly applied to a wide variety of models containing RNN architectures.