RI: Small: Visual Reasoning and Self-questioning for Explainable Visual Question Answering

视觉问答(VQA)旨在用自然语言回答与给定图像相关的问题，目前仍处于起步阶段。目前的方法缺乏灵活性和普遍性，不能在不经过培训的情况下处理不同的问题。因此，开发可解释的VQA(或X-VQA)是可取的，它除了提供答案之外，还可以用自然语言对其推理进行解释。这需要将计算机视觉、自然语言和知识表示相结合，这是一项极具挑战性的任务。通过探索X-VQA，本项目推进和丰富了计算机视觉、图像理解、视觉语义分析、机器学习和知识表示的基础知识。它还极大地促进了包括视觉聊天机器人、视觉检索和推荐以及人机交互在内的广泛应用。这项研究还通过课程开发、学生培训和知识传播为教育做出贡献。它包括与K-12学生的互动，以获得参与和研究机会。本研究的主要目的是开发一种具有坚实理论基础和有效方法的新型计算模型，以便为X-VQA的可视化推理提供解释。这项具有挑战性的任务涉及许多基本方面，需要将视野、语言、学习和知识结合起来。本课题主要研究内容包括：(1)建立了X-VQA的统一计算模型及其理论基础。该模型集成了领域知识和视觉观察来进行推理：从不完整和不准确的视觉观察中可以推断出什么以及如何推断隐藏的事实；如何表示视觉观察、隐藏的事实和领域知识以实现高效的问题回答；以及问题回答如何可以扩展。这些关键问题的研究为X-VQA奠定了基础；(2)提出了问题驱动、任务型视觉观察的新模式。在回答问题之前收集所有的视觉观察是效率低下的。愿景需要以问题为导向，以任务为导向。该项目追求一种新的问题交互、视觉推理和视觉观察的模式，从而自动将注意力引导到图像中与问题相关的方面；(3)用于培训X-VQA代理的自我提问的创新方法。单纯基于问答数据的培训对X-VQA来说是不可行的，因为它无法提供对答案的解释和洞察。这个项目追求一种新的自我质疑的方法，在这种方法中，VQA代理也可以产生和提出问题。它调查了如何将自我提问与强化学习相结合，以及如何处理各种问题以提高X-VQA的可扩展性；以及(4)关于X-VQA的可靠案例研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Visual question answering (VQA), aiming to answer a question in natural language related to a given image, is still in its infancy. Current approaches lack flexibility and generalizability to handling diverse questions without training. It is therefore desirable to explorep explainable VQA (or X-VQA) that can provide explanations of its reasoning in natural language in addition to answers. This requires integrating computer vision, natural language, and knowledge representation, and it is an incredibly challenging task. By exploring X-VQA this project advances and enriches the fundamental computer vision, image understanding, visual semantic analysis, machine learning, and knowledge representation. And it also greatly facilitates a wide range of applications including visual chatbots, visual retrieval and recommendation, and human-computer interaction. This research also contributes to education through curriculum development, student training, and knowledge dissemination. It includes interactions with K-12 students for participation and research opportunities. The major goal of this research is to develop a novel computational model with solid theoretical foundation and effective methods, to facilitate X-VQA that provides explanations of its visual reasoning. This challenging task involves many fundamental aspects and needs to integrate vision, language, learning and knowledge. This project focuses on: (1) A unified computational model of X-VQA and its theoretical foundation. This model integrates domain knowledge and visual observations for reasoning: what and how hidden facts can be inferred from incomplete and inaccurate visual observations; how visual observation, hidden facts, and domain knowledge can be represented for efficient question answering; and how the question answering can be scalable. The study of these critical issues creates the foundation for X-VQA; (2) A new model for question-driven task-oriented visual observation. It is inefficient to collect all visual observations before answering a question. Vision needs to be question-driven and task-oriented. This project pursues a new model for the interaction of questions, visual reasoning and visual observation, so as to automatically steer attention to the question-related aspects of an image; (3) An innovative approach to self-questioning for training X-VQA agents. Training simply based on question-answer data is not viable for X-VQA, as it is unable to provide explanations for and insights into the answer. This project pursues a novel approach to self-questioning, in which the VQA agents can also generate and ask questions. It investigates how self-questioning can be combined with reinforcement learning, and how it can deal with versatile questions to improve the scalability of X-VQA; and (4) A solid case study on X-VQA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.