Multi-Target Embodied Question Answering
Multi-Target Embodied Question Answering
复制标题
DOI：
10.1109/cvpr.2019.00647
复制
发表时间：
2019-04
期刊：
影响因子：
--
通讯作者：
Licheng Yu;Xinlei Chen;Georgia Gkioxari;Mohit Bansal;Tamara L. Berg;Dhruv Batra
中科院分区：
文献类型：
--
作者：
Licheng Yu;Xinlei Chen;Georgia Gkioxari;Mohit Bansal;Tamara L. Berg;Dhruv Batra
具身问答（Embodied Question answer， EQA）是一项相对较新的任务，它要求agent根据自我中心感知回答有关其环境的问题。b[8]中介绍的EQA做出了一个基本假设，即每个问题，例如“汽车是什么颜色的？”，都只有一个目标（“汽车”）被询问。这个假设直接限制了代理的能力。我们提出了一种广义的多目标EQA （MT-EQA）。具体来说，我们研究有多个目标的问题，比如“卧室里的梳妆台比厨房里的烤箱大吗？”，在回答问题之前，智能体必须导航到多个位置（“卧室里的梳妆台”，“厨房里的烤箱”），并进行比较推理（“梳妆台”比“烤箱”大）。这样的问题需要在代理中开发全新的模块或组件。为了解决这个问题，我们提出了一个由程序生成器、控制器、导航器和VQA模块组成的模块化体系结构。程序生成器将给定的问题转换为顺序的可执行子程序；导航器引导代理到与导航相关的子程序相关的多个位置；控制器学习在其路径上选择相关的观测值。然后将这些观察结果馈送到VQA模块以预测答案。我们对每个模型组件进行了详细的分析，并表明我们的联合模型可以在很大程度上优于以前的方法和强基线。
Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA as introduced in [8] makes the fundamental assumption that every question, e.g., ``what color is the car?", has exactly one target (``car") being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA -- Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as ``Is the dresser in the bedroom bigger than the oven in the kitchen?", where the agent has to navigate to multiple locations (``dresser in bedroom", ``oven in kitchen") and perform comparative reasoning (``dresser" bigger than ``oven") before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin.