Language Instructed Reinforcement Learning for Human-AI Coordination-猫眼课题宝

DOI：

10.48550/arxiv.2304.07297

发表时间：

2023-04

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Hengyuan Hu;Dorsa Sadigh

人工智能的基本任务之一是产生与人类良好协调的代理。这个问题具有挑战性，特别是在缺乏高质量人类行为数据的领域，因为多智能体强化学习（RL）通常会收敛到与人类偏好的不同的平衡。我们提出了一个新颖的框架，instructRL，它使人类能够通过自然语言指令指定他们期望人工智能合作伙伴采取什么样的策略。我们使用预训练的大型语言模型来生成以人类指令为条件的先验策略，并使用先验策略来规范 RL 目标。这导致强化学习代理收敛到符合人类偏好的平衡。我们证明，instructRL 收敛于类人策略，满足概念验证环境中给定的指令以及具有挑战性的 Hanabi 基准。最后，我们表明，了解语言指令可以显着提高 Hanabi 中人类评估中的人类与人工智能的协调性能。

One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.

Language Instructed Reinforcement Learning for Human-AI Coordination

文献摘要

相似文献