Cautious Actor-Critic
Cautious Actor-Critic
复制标题
DOI：
--
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Lingwei Zhu;Toshinori Kitamura;Takamitsu Matsubara
中科院分区：
文献类型：
--
作者：
Lingwei Zhu;Toshinori Kitamura;Takamitsu Matsubara
非策略学习的振荡性能和ACTOR CRITIC（AC）设置中的持续错误要求算法可以保守地学习以更好地适应稳定性关键应用。在本文中，我们提出了一种新的非策略AC算法谨慎的行动者-评论家（CAC）。谨慎的名字来自双重保守的性质，我们利用经典的政策插值从保守的政策迭代的演员和保守的价值迭代的熵正则化的批评。我们的主要观察是熵正则化的评论家促进和简化了笨拙的插值演员更新，同时仍然确保强大的政策改进。我们比较CAC的一组具有挑战性的连续控制问题的最先进的AC方法，并证明CAC实现可比的性能，同时显着稳定的学习。
The oscillating performance of off-policy learning and persisting errors in the actor-critic (AC) setting call for algorithms that can conservatively learn to suit the stability-critical applications better. In this paper, we propose a novel off-policy AC algorithm cautious actor-critic (CAC). The name cautious comes from the doubly conservative nature that we exploit the classic policy interpolation from conservative policy iteration for the actor and the entropy-regularization of conservative value iteration for the critic. Our key observation is the entropy-regularized critic facilitates and simplifies the unwieldy interpolated actor update while still ensuring robust policy improvement. We compare CAC to state-of-the-art AC methods on a set of challenging continuous control problems and demonstrate that CAC achieves comparable performance while significantly stabilizes learning.