RI: Small: Learning Dynamics and Evolution towards Cognitive Understanding of Videos

人类智能的一项基本能力是能够通过观看教学视频来学习行动。这种能力体现在对教学程序的抽象和总结中，以及在回答视频中发生的一些问题时，如“为什么”和“如何”。该项目旨在建立能够很好地执行上述任务的计算模型，这些任务需要对场景中的对象、动作和属性进行常规识别，并对其中的任何关系进行更高阶的推理。这里，高阶推理指的是不能用直接观察立即回答的推理，因此需要更强的语义。开发的技术将使其他领域的许多应用成为可能，例如多媒体(视频索引和检索)、机器人(为什么和如何提问的推理能力)和医疗保健(为视障人士提供的辅助设备)。此外，该项目将通过让代表性不足的群体参与研究活动、将研究成果纳入教学课程以及向当地K-12社区开展外联活动，促进教育和多样性。本研究将开发一个在理解网络教学视频时执行高阶推理的框架，以便在该框架中设计的模型不仅能够发现和说明构成教学事件的过程，而且能够回答诸如为什么和如何发生的问题。该框架建立在视频故事图的基础上，对动态(不同尺度上动作的组合)和演化(对象状态和属性的变化)进行建模，并支持在深度学习单元上进行更高阶推理，并将外部知识图整合到统一的框架中。将探索提取此类视频故事图表并使用它们来发现、字幕程序和执行问答的方法。该项目的预期成果包括：一个用于构建和执行对视频故事图表的推理并纳入外部知识的软件包；一个处理用户上传的教学视频的网络部署系统；以及一个带有程序和问答注释的大型视频数据集。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

A fundamental capability of human intelligence is being able to learn to act by watching instructional videos. Such capability is reflected in abstraction and summarization of the instructional procedures as well as in answering questions such as "why" and "how" something happened in the video. This project aims to build computational models that are able to perform well in above tasks, which require, beyond the conventional recognition of objects, actions and attributes in the scene, the higher-order inference of any relations therein. Here, the higher-order inference refers to inference that cannot be answered immediately by direct observations and thus requires stronger semantics. The developed technology will enable many applications in other fields, e.g., multimedia (video indexing and retrieval), robotics (reasoning capability of why and how questions), and healthcare (assistive devices for visually impaired people). In addition, the project will contribute to education and diversity by involving underrepresented groups in research activities, integrating research results into teaching curriculum, and conducting outreach activities to local K-12 communities. The research will develop a framework to perform higher-order inference in understanding web instructional videos, such that models devised in this framework are capable of not only discovering and captioning procedures that constitute the instructional event but also answering questions such as why and how something happened. The framework is built on a video story graph that models the dynamics (the composition of actions at different scales) and evolution (the change in object states and attributes), and it supports higher-order inference upon deep learning units and incorporation of external knowledge graph in a unified framework. Methodologies to extract such video story graphs and use them to discover, caption procedures and perform question-answering will be explored. Expected outcomes of this project include: a software package for constructing and performing inference on video story graphs and incorporating external knowledge; a web-deployed system to process user-uploaded instructional videos; and a large video dataset with procedure and question-answering annotations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.