CAREER: Visual Question Answering (VQA)

该项目解决了可视化问题查询（VQA）的问题。给定图像和关于图像的自由形式的自然语言问题（例如，“这是什么店？有多少人在排队？“，“过马路安全吗？”），机器的任务是自动产生一个简洁、准确、形式自由的自然语言答案（“面包店”、“5”、“是”）。VQA直接适用于各种具有高度社会影响的应用，这些应用涉及人类从视觉数据中获取情境相关信息;人类和机器必须合作从图片中提取信息。例子包括帮助视力受损的用户了解他们的周围环境，分析师根据大量的监控做出决策，以及与机器人互动。该项目有可能从根本上改善视障用户的日常生活方式，并彻底改变整个社会与视觉数据的交互方式。这项研究使VQA代表的不是一个单一的狭义定义的问题（例如，图像分类），而是语义场景理解问题和相关研究方向的丰富谱。VQA中的每个问题都可能位于这个频谱上的不同点：从直接映射到现有的经过充分研究的计算机视觉问题的问题（“这个房间叫什么？=室内场景识别）一直到需要在知识库上综合视觉（场景）、语言（语义）和推理（理解）的问题（“后排可乐瓶旁边的披萨看起来像素食吗？").因此，该工作映射到沿该谱沿着的一系列航路点。该研究计划的动机是从各种角度解决VQA问题，该研究计划正在（i）纯计算机视觉（ii）整合视觉+语言（iii）整合视觉+语言+常识（iv）构建可解释的模型（v）结合方法组合中生成新的数据集，知识和技术。此外，还在以下方面做出了新的贡献：（a）训练机器保持好奇心，并积极提出问题进行学习（B）使用VQA作为一种模态，以了解比现有注释模态所允许的更多的视觉世界;（c）训练机器知道它知道什么，不知道什么。

This project addresses the problem of Visual Question Answering (VQA). Given an image and a free-form natural language question about the image (e.g., "What kind of store is this?", "How many people are waiting in the queue?", "Is it safe to cross the street?"), the machine's task is to automatically produce a concise, accurate, free-form, natural language answer ("bakery", "5", "Yes"). VQA is directly applicable to a variety of applications of high societal impact that involve humans eliciting situationally-relevant information from visual data; where humans and machines must collaborate to extract information from pictures. Examples include aiding visually-impaired users in understanding their surroundings, analysts in making decisions based on large quantities of surveillance, and interacting with a robot. This project has the potential to fundamentally improve the way visually-impaired users live their daily lives, and revolutionize how society at large interacts with visual data. This research enables that VQA represents not a single narrowly-defined problem (e.g., image classification) but rather a rich spectrum of semantic scene understanding problems and associated research directions. Each question in VQA may lie at a different point on this spectrum: from questions that directly map to existing well-studied computer-vision problems ("What is this room called?" = indoor scene recognition) all the way to questions that require an integrated approach of vision (scene), language (semantics), and reasoning (understanding) over a knowledge base ("Does the pizza in the back row next to the bottle of Coke seem vegetarian?"). Consequently, this work maps to a sequence of waypoints along this spectrum. Motivated by addressing VQA from a variety of perspectives, this research program is generating new datasets, knowledge, and techniques in (i) pure computer vision (ii) integrating vision + language (iii) integrating vision + language + common sense (iv) building interpretable models and (v) combining a portfolio of methods. In addition, novel contributions are being made to (a) training the machine to be curious and actively ask questions to learn (b) using VQA as a modality to learn more about the visual world than what existing annotation modalities allow and (c) training the machine to know what it knows and what it does not.