RI: Small: Visual Reasoning and Self-questioning for Explainable Visual Question Answering

RI：小：视觉推理和自我质疑以实现可解释的视觉问答

• 批准号: 2007613
• 负责人: Ying Wu
• 金额: $ 46.92万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007613&HistoricalAwards=false
• 关键词: RI; Small; Visual; Reasoning; Self

Visual question answering (VQA), aiming to answer a question in natural language related to a given image, is still in its infancy. Current approaches lack flexibility and generalizability to handling diverse questions without training. It is therefore desirable to explorep explainable VQA (or X-VQA) that can provide explanations of its reasoning in natural language in addition to answers. This requires integrating computer vision, natural language, and knowledge representation, and it is an incredibly challenging task. By exploring X-VQA this project advances and enriches the fundamental computer vision, image understanding, visual semantic analysis, machine learning, and knowledge representation. And it also greatly facilitates a wide range of applications including visual chatbots, visual retrieval and recommendation, and human-computer interaction. This research also contributes to education through curriculum development, student training, and knowledge dissemination. It includes interactions with K-12 students for participation and research opportunities. The major goal of this research is to develop a novel computational model with solid theoretical foundation and effective methods, to facilitate X-VQA that provides explanations of its visual reasoning. This challenging task involves many fundamental aspects and needs to integrate vision, language, learning and knowledge. This project focuses on: (1) A unified computational model of X-VQA and its theoretical foundation. This model integrates domain knowledge and visual observations for reasoning: what and how hidden facts can be inferred from incomplete and inaccurate visual observations; how visual observation, hidden facts, and domain knowledge can be represented for efficient question answering; and how the question answering can be scalable. The study of these critical issues creates the foundation for X-VQA; (2) A new model for question-driven task-oriented visual observation. It is inefficient to collect all visual observations before answering a question. Vision needs to be question-driven and task-oriented. This project pursues a new model for the interaction of questions, visual reasoning and visual observation, so as to automatically steer attention to the question-related aspects of an image; (3) An innovative approach to self-questioning for training X-VQA agents. Training simply based on question-answer data is not viable for X-VQA, as it is unable to provide explanations for and insights into the answer. This project pursues a novel approach to self-questioning, in which the VQA agents can also generate and ask questions. It investigates how self-questioning can be combined with reinforcement learning, and how it can deal with versatile questions to improve the scalability of X-VQA; and (4) A solid case study on X-VQA.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视觉问答（VQA），旨在回答与给定图像相关的自然语言问题，仍处于起步阶段。 目前的方法缺乏灵活性和通用性，在没有培训的情况下处理不同的问题。因此，希望探索可解释的VQA（或X-VQA），除了答案之外，还可以用自然语言解释其推理。这需要集成计算机视觉、自然语言和知识表示，这是一项极具挑战性的任务。通过探索X-VQA，该项目推进和丰富了基础计算机视觉，图像理解，视觉语义分析，机器学习和知识表示。它也极大地促进了广泛的应用，包括视觉聊天机器人，视觉检索和推荐以及人机交互。这项研究也有助于通过课程开发，学生培训和知识传播的教育。它包括与K-12学生的参与和研究机会的互动。 本研究的主要目标是建立一个新的计算模型，具有坚实的理论基础和有效的方法，以促进X-VQA，提供其视觉推理的解释。这项具有挑战性的任务涉及许多基本方面，需要将视觉，语言，学习和知识结合起来。本课题主要研究内容如下：（1）X-VQA统一计算模型及其理论基础。该模型集成了领域知识和视觉观察的推理：什么以及如何隐藏的事实可以推断出不完整和不准确的视觉观察;视觉观察，隐藏的事实和领域知识可以表示为有效的问题回答;以及问题回答如何可扩展。这些关键问题的研究为X-VQA的研究奠定了基础;（2）提出了一种新的问题驱动的任务导向视觉观察模型。在回答问题之前收集所有视觉观察结果是低效的。愿景需要以问题为驱动，以任务为导向。该项目追求一种新的模型的互动问题，视觉推理和视觉观察，从而自动引导注意力的问题相关的方面的图像;（3）一种创新的方法来自我提问的训练X-VQA代理。简单地基于问答数据的训练对于X-VQA来说是不可行的，因为它无法为答案提供解释和见解。该项目追求一种新颖的自我提问方法，其中VQA代理也可以生成和提问。它研究了自我提问如何与强化学习相结合，以及如何处理多功能问题以提高X-VQA的可扩展性;（4）X-VQA的坚实案例研究。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Unsupervised Depth Completion and Denoising for RGB-D Sensors

RGB-D 传感器的无监督深度补全和去噪

• DOI: 10.1109/icra46639.2022.9812392
• 发表时间: 2022
• 期刊: 2022 International Conference on Robotics and Automation (ICRA
• 作者: Fan, Lei;Li, Yunxuan;Jiang, Chen;Wu, Ying
• 通讯作者: Wu, Ying

Morphable Detector for Object Detection on Demand

• DOI: 10.1109/iccv48922.2021.00473
• 发表时间: 2021-10
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Xiangyun Zhao;Xu Zou;Ying Wu

Avoiding Lingering in Learning Active Recognition by Adversarial Disturbance

• DOI: 10.1109/wacv56688.2023.00459
• 发表时间: 2023-01
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Lei Fan;Ying Wu

Contrastive Learning for Label Efficient Semantic Segmentation

• DOI: 10.1109/iccv48922.2021.01045
• 发表时间: 2020-12
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Xiangyu Zhao;Raviteja Vemulapalli;P. A. Mansfield;Boqing Gong;Bradley Green;Lior Shapira;Ying Wu

Temporal Feature Enhancement Dilated Convolution Network for Weakly-supervised Temporal Action Localization

• DOI: 10.1109/wacv56688.2023.00597
• 发表时间: 2023-01
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Jianxiong Zhou;Ying Wu