RI: Small: Visual Reasoning and Self-questioning for Explainable Visual Question Answering

视觉问答（VQA），旨在用自然语言回答与给定图像相关的问题，仍处于起步阶段。目前的方法缺乏灵活性和通用性，无法在未经培训的情况下处理各种问题。因此，探索可解释的VQA（或X-VQA）是可取的，除了答案之外，它还可以用自然语言提供其推理的解释。这需要集成计算机视觉、自然语言和知识表示，这是一项极具挑战性的任务。通过探索X-VQA，该项目推进并丰富了基础的计算机视觉、图像理解、视觉语义分析、机器学习和知识表示。它也极大地促进了广泛的应用，包括视觉聊天机器人，视觉检索和推荐，人机交互。本研究亦透过课程发展、学生训练及知识传播，为教育作出贡献。它包括与K-12学生的参与和研究机会的互动。本研究的主要目标是开发一种具有坚实理论基础和有效方法的新型计算模型，以促进X-VQA提供其视觉推理的解释。这项具有挑战性的任务涉及许多基本方面，需要整合视觉、语言、学习和知识。本课题重点研究：(1)X-VQA统一计算模型及其理论基础。该模型集成了领域知识和视觉观察来进行推理：从不完整和不准确的视觉观察中可以推断出隐藏的事实是什么以及如何推断出来的；如何将视觉观察、隐藏事实和领域知识表示为有效的问答；以及问题的回答如何可以扩展。对这些关键问题的研究为X-VQA奠定了基础；(2)问题驱动任务导向视觉观察新模型。在回答问题之前收集所有的视觉观察是低效的。愿景需要问题驱动和任务导向。本项目追求问题、视觉推理和视觉观察互动的新模式，从而自动将注意力引导到图像中与问题相关的方面；(3)创新X-VQA座席自我提问方式。单纯基于问答数据的训练对于X-VQA来说是不可行的，因为它无法提供答案的解释和见解。该项目追求一种新颖的自我质疑方法，其中VQA代理也可以生成和提问问题。它研究了如何将自我质疑与强化学习相结合，以及如何处理通用问题以提高X-VQA的可扩展性；(4) X-VQA的实例研究。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Visual question answering (VQA), aiming to answer a question in natural language related to a given image, is still in its infancy. Current approaches lack flexibility and generalizability to handling diverse questions without training. It is therefore desirable to explorep explainable VQA (or X-VQA) that can provide explanations of its reasoning in natural language in addition to answers. This requires integrating computer vision, natural language, and knowledge representation, and it is an incredibly challenging task. By exploring X-VQA this project advances and enriches the fundamental computer vision, image understanding, visual semantic analysis, machine learning, and knowledge representation. And it also greatly facilitates a wide range of applications including visual chatbots, visual retrieval and recommendation, and human-computer interaction. This research also contributes to education through curriculum development, student training, and knowledge dissemination. It includes interactions with K-12 students for participation and research opportunities. The major goal of this research is to develop a novel computational model with solid theoretical foundation and effective methods, to facilitate X-VQA that provides explanations of its visual reasoning. This challenging task involves many fundamental aspects and needs to integrate vision, language, learning and knowledge. This project focuses on: (1) A unified computational model of X-VQA and its theoretical foundation. This model integrates domain knowledge and visual observations for reasoning: what and how hidden facts can be inferred from incomplete and inaccurate visual observations; how visual observation, hidden facts, and domain knowledge can be represented for efficient question answering; and how the question answering can be scalable. The study of these critical issues creates the foundation for X-VQA; (2) A new model for question-driven task-oriented visual observation. It is inefficient to collect all visual observations before answering a question. Vision needs to be question-driven and task-oriented. This project pursues a new model for the interaction of questions, visual reasoning and visual observation, so as to automatically steer attention to the question-related aspects of an image; (3) An innovative approach to self-questioning for training X-VQA agents. Training simply based on question-answer data is not viable for X-VQA, as it is unable to provide explanations for and insights into the answer. This project pursues a novel approach to self-questioning, in which the VQA agents can also generate and ask questions. It investigates how self-questioning can be combined with reinforcement learning, and how it can deal with versatile questions to improve the scalability of X-VQA; and (4) A solid case study on X-VQA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.