RI: Small: Visual How: Task Understanding and Description in the Real World

解决问题是人类通过进化和经验而发展的一种与生俱来的能力。与能够解决一般和复杂问题的人类智能相比，目前的人工智能系统只能在狭窄和结构化的任务中表现良好。随着弥合这一差距的总体目标，该项目开发的人工智能系统，可以理解一般的现实世界的任务（例如，如何搭建帐篷？如何教孩子们园艺？如何在伦敦旅行？）并通过逐步的语言和视觉指导提出解决方案。它将允许在一般和复杂的情况下解决现实世界的任务，从而产生更像人类的人工智能。最终，该项目将朝着人工通用智能迈出一步。该项目将提供一个公开的数据集，计算模型的框架，和一个移动的应用原型。此外，该项目将支持综合研究和教育，重点是通过K-12外展，代表性不足和本科生指导，以及课程开发来增加少数民族的参与。问题的一般性和复杂性要求能够理解任务的视觉和文本内容，推理与任务相关的知识，并生成关于如何完成任务的逐步多模态描述。本项目旨在通过三项任务实现这些目标。首先，生成一个新的数据集，其中包含各种真实世界的任务和解决方案，并具有关键语义和任务结构的丰富注释，以指导多模态注意力和结构推理。第二，开发一个新的框架，在该框架中导出了一系列模型，用于可解释的VisualHow学习，以理解视觉文本内容并生成完成现实世界任务的步骤。第三，开发新的方法来概括模型的知识，并在移动的平台上验证它们，以帮助人们在现实世界中的应用。实现这些目标不仅将为解决现实世界问题带来新的视觉语言任务和计算方法，还将推动可解释和可推广的人工智能模型和系统的开发创新。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Problem solving is an innate capability that humans develop through evolution and experience. Compared to human intelligence that can solve general and complex problems, current AI systems only perform well in narrow and structured tasks. With the overarching goal of bridging this gap, this project develops AI systems that can understand general real-world tasks (e.g., How to set up a tent? How to teach kids to garden? How to travel in London?) and come up with solutions with step-by-step language and visual guidance. It will allow for real-world tasks to be solved even in general and complex circumstances, resulting in more human-like AI. Ultimately, the project will take a step forward toward artificial general intelligence. The project will provide a publicly available dataset, a framework of computational models, and a mobile application prototype. Furthermore, this project will support integrated research and education with a focus on increasing minority participation through K-12 outreach, underrepresented and undergraduate mentoring, and curriculum development.This project proposes a VisualHow problem that represents a rich spectrum of real-world tasks. The generality and complexity of the problem call for capabilities to understand the visual and textual contents of the task, reason with knowledge relevant to the task, and generate step-by-step multimodal descriptions about how the task can be completed. This project aims to achieve these goals in three tasks. First, generate a new dataset with diverse and real-world tasks and solutions, with rich annotations of key semantics and task structures to guide the multimodal attention and structural reasoning. Second, develop a novel framework in which a series of models are derived for explainable VisualHow learning to understand the visual-textual contents and generate steps to complete real-world tasks. Third, develop novel methods to generalize the models with knowledge and validate them on mobile platforms to assist people in real-world applications. Achieving these goals will not only lead to new vision-language tasks and computational methods for real-world problem solving, but also spur innovations in the development of explainable and generalizable AI models and systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.