RI: Small: Visual How: Task Understanding and Description in the Real World

RI：小：视觉方式：现实世界中的任务理解和描述

• 批准号: 2143197
• 负责人: Qi Zhao
• 金额: $ 26.22万
• 依托单位: University of Minnesota-Twin Cities
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2143197&HistoricalAwards=false
• 关键词: RI; Small; Visual; How; Task

Problem solving is an innate capability that humans develop through evolution and experience. Compared to human intelligence that can solve general and complex problems, current AI systems only perform well in narrow and structured tasks. With the overarching goal of bridging this gap, this project develops AI systems that can understand general real-world tasks (e.g., How to set up a tent? How to teach kids to garden? How to travel in London?) and come up with solutions with step-by-step language and visual guidance. It will allow for real-world tasks to be solved even in general and complex circumstances, resulting in more human-like AI. Ultimately, the project will take a step forward toward artificial general intelligence. The project will provide a publicly available dataset, a framework of computational models, and a mobile application prototype. Furthermore, this project will support integrated research and education with a focus on increasing minority participation through K-12 outreach, underrepresented and undergraduate mentoring, and curriculum development.This project proposes a VisualHow problem that represents a rich spectrum of real-world tasks. The generality and complexity of the problem call for capabilities to understand the visual and textual contents of the task, reason with knowledge relevant to the task, and generate step-by-step multimodal descriptions about how the task can be completed. This project aims to achieve these goals in three tasks. First, generate a new dataset with diverse and real-world tasks and solutions, with rich annotations of key semantics and task structures to guide the multimodal attention and structural reasoning. Second, develop a novel framework in which a series of models are derived for explainable VisualHow learning to understand the visual-textual contents and generate steps to complete real-world tasks. Third, develop novel methods to generalize the models with knowledge and validate them on mobile platforms to assist people in real-world applications. Achieving these goals will not only lead to new vision-language tasks and computational methods for real-world problem solving, but also spur innovations in the development of explainable and generalizable AI models and systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

解决问题是人类通过进化和经验发展的先天能力。与可以解决一般和复杂问题的人类智能相比，当前的AI系统仅在狭窄且结构化的任务中表现良好。凭借弥合这一差距的总体目标，该项目开发了可以理解一般现实世界任务的AI系统（例如，如何建立帐篷？如何教孩子们去伦敦旅行？）并通过逐步的语言和视觉指导提出解决方案。即使在一般和复杂的环境中，也可以解决现实世界的任务，从而导致更类似人类的AI。最终，该项目将迈向人工通用情报。该项目将提供公开可用的数据集，计算模型的框架和移动应用程序原型。此外，该项目将支持综合的研究和教育，重点是通过K-12外展，代表性不足和本科指导以及课程开发来增加少数群体的参与。该项目提出了一个视觉速度问题，代表着丰富的现实世界任务。问题的一般性和复杂性要求能够理解任务的视觉和文本内容，与任务相关的知识的原因，并生成有关如何完成任务的逐步多模式描述。该项目旨在在三个任务中实现这些目标。首先，生成一个具有多种多样和现实世界的任务和解决方案的新数据集，并使用关键语义和任务结构的丰富注释来指导多模式的关注和结构推理。其次，开发一个新颖的框架，在该框架中，一系列模型被得出可解释的视觉方式学习，以了解视觉文本内容并生成步骤以完成现实世界任务。第三，开发新的方法来概括具有知识的模型，并在移动平台上验证它们，以帮助人们进行现实世界中的应用程序。实现这些目标不仅会导致现实世界中解决问题的新型任务和计算方法，而且还将刺激创新的创新，以开发可解释且可推广的AI模型和系统。这项奖项反映了NSF的法定任务，并被认为是通过基金会的智力功能和广泛的影响来评估CRETIRIA的评估，以评估值得通过评估。

项目成果

Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning

• DOI: 10.48550/arxiv.2303.10482
• 发表时间: 2023-03
• 期刊: ArXiv
• 作者: Shi Chen;Qi Zhao

VisualHow: Multimodal Problem Solving

• DOI: 10.1109/cvpr52688.2022.01518
• 发表时间: 2022-06
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Jinhui Yang;Xianyu Chen;Ming Jiang;Shi Chen;Louis Wang;Qi Zhao