III: Medium: Learning Multimodal Knowledge about Entities and Events

关于世界的日常知识是智能信息处理和推理的必要条件。由于了解世界如何运作的日常功能知识，人们可以阅读文本的字里行间，并超越图像中可见的内容。这项研究的主要目标是开发学习算法，能够从大规模多模式网络数据中自动获取以实体和事件为中心的知识。实体知识包括关于物体和人的广泛的物理和概念知识，包括它们的属性、它们的相对差异以及它们之间的逻辑关系。事件知识侧重于通过子事件和事件参与者之间的层次和时间关系组织的有关人们生活中日常事件的结构知识。总之，由此产生的知识将成为在自然语言处理和计算机视觉交叉点实现强大的人工智能系统的关键一步，该系统可以理解和推理非结构化多模态信息。这项研究的潜在影响包括针对视障人士的交互式辅助系统和多模式教育界面。该项目研究多模态知识提取作为一种新的研究范式，将自然语言处理中的相关方法（例如信息提取、文本蕴涵和框架语义）与计算机视觉的最新进展联系起来。获取常识知识的关键挑战之一是克服报告偏见，即人们不陈述显而易见的事实。因此，该项目开发了基于图的集体推理的新学习算法，可以推理出不言而喻的知识，系统地影响人们用语言、图像和视频描述世界的方式。此外，该项目还开发了用于视觉语义解析和事件识别的新模型，通过指定事件的各种结构组件（例如参与者、对象、位置、工具、意图和目标）来概括现有的活动识别研究。所学到的知识和表达将通过多种应用程序进行验证，包括多模式问答和基础语言理解。

Everyday knowledge about the world is a necessary condition for intelligent information processing and reasoning. People can read between the lines in text and see beyond what are visible in images because of everyday functional knowledge about how the world works. The primary goal of this research is to develop learning algorithms that can automatically acquire such knowledge, centered around entities and events, from large-scale multimodal web data. Entity knowledge includes a broad range of physical and conceptual knowledge about objects and people, including their attributes, their relative differences, and logical relations among them. Event knowledge focuses on structural knowledge about everyday events in people's lives organized through hierarchical and temporal relations among sub-events and the event participants. Together, the resulting knowledge will be a critical step forward to enable robust AI systems at the intersection between natural language processing and computer vision that can understand and reason about unstructured multimodal information. The potential impact of this research includes interactive assistive systems for the visually-impaired and multimodal educational interfaces. This project investigates multimodal knowledge extraction as a new research paradigm drawing connections between relevant methods in natural language processing such as information extraction, textual entailments, and frame semantics with recent advances in computer vision. One of the critical challenges in commonsense knowledge acquisition is to overcome reporting bias, i.e., people do not state the obvious. Therefore, this project develops new learning algorithms based on a graph-based collective inference that can reason about unspoken knowledge that systematically influences the way people describe the world in language, images, and videos. In addition, this project develops new models for visual semantic parsing and event recognition, which generalize existing studies on activity recognition by specifying various structural components of events such as actors, objects, locations, tools, intents, and goals. The learned knowledge and representation will be validated through several applications including multimodal question answering and grounded language understanding.