A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-07
期刊：
影响因子：
--
通讯作者：
O. Marschall;Kyunghyun Cho;Cristina Savin
中科院分区：
文献类型：
--
作者：
O. Marschall;Kyunghyun Cho;Cristina Savin
我们提出了一个框架，以全面总结许多最近的结果，在有效的和/或生物学上合理的在线训练的递归神经网络（RNN）。该框架根据几个标准组织算法：（a）过去与面向未来，（B）张量结构，（c）随机与确定性，以及（d）封闭形式与数值。这些轴揭示了在线学习的几个最新进展之间潜在的概念联系。此外，我们为他们的成功程度提供了新的数学直觉。在两个合成任务上测试各种算法表明，根据我们的标准，性能聚类。虽然梯度对齐也观察到类似的聚类，但与精确方法对齐并不能单独解释最终性能，特别是对于随机算法。这表明需要更好的比较指标。
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.