Riemannian Adaptive Optimization Algorithm and its Application to Natural Language Processing
Riemannian Adaptive Optimization Algorithm and its Application to Natural Language Processing
复制标题
DOI：
10.1109/tcyb.2021.3049845
复制
发表时间：
2020-04
影响因子：
11.8
通讯作者：
Hiroyuki Sakai;H. Iiduka
中科院分区：
文献类型：
--
作者：
Hiroyuki Sakai;H. Iiduka
本文提出了一种黎曼自适应优化算法来优化深度神经网络的参数。该算法是欧氏空间中AMSGrad和黎曼流形上RAMSGrad的推广。该算法有助于解决影响RAMSGrad的两个问题。首先，它可以直接解决黎曼随机优化问题，而RAMSGrad只能实现低遗憾。另一个是它可以使用恒定的学习率，这使得它在实践中可以实现。此外，我们将所提出的算法应用于庞加莱嵌入，将WordNet名词的传递闭包嵌入到双曲空间的庞加莱球模型中。数值实验表明，无论学习率的初始值，我们的算法稳定地收敛到最优解，收敛速度比现有的算法。
This article proposes a Riemannian adaptive optimization algorithm to optimize the parameters of deep neural networks. The algorithm is an extension of both AMSGrad in Euclidean space and RAMSGrad on a Riemannian manifold. The algorithm helps to resolve two issues affecting RAMSGrad. The first is that it can solve the Riemannian stochastic optimization problem directly, in contrast to RAMSGrad which only achieves a low regret. The other is that it can use constant learning rates, which makes it implementable in practice. Additionally, we apply the proposed algorithm to Poincaré embeddings that embed the transitive closure of the WordNet nouns into the Poincaré ball model of hyperbolic space. Numerical experiments show that regardless of the initial value of the learning rate, our algorithm stably converges to the optimal solution and converges faster than the existing algorithms.