Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
复制标题
DOI：
10.18653/v1/d18-1287
复制
发表时间：
2018-09
期刊：
影响因子：
--
通讯作者：
Dipendra Misra;Andrew Bennett;Valts Blukis;Eyvind Niklasson;Max Shatkhin;Yoav Artzi
中科院分区：
文献类型：
--
作者：
Dipendra Misra;Andrew Bennett;Valts Blukis;Eyvind Niklasson;Max Shatkhin;Yoav Artzi
我们提议将指令执行分解为目标预测和动作生成。我们设计了一个模型，该模型使用LINGUNET（一种语言条件图像生成网络）将原始视觉观察映射到目标，然后生成完成这些目标所需的动作。我们的模型仅通过演示进行训练，无需外部资源。为了评估我们的方法，我们引入了两个指令遵循的基准：LANI，一个导航任务；以及CHAI，其中智能体执行家庭指令。我们的评估展示了我们模型分解的优势，并说明了我们新基准所带来的挑战。
We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.