III: Medium: Learning Multimodal Knowledge about Entities and Events

关于世界的日常知识是智能信息处理和推理的必要条件。人们能够读懂文字的字里行间，看到图像之外的东西，这是因为人们对世界如何运转的日常功能知识。本研究的主要目标是开发能够从大规模多模态网络数据中自动获取以实体和事件为中心的知识的学习算法。实体知识包括关于物体和人的广泛的物理和概念知识，包括它们的属性、它们的相对差异以及它们之间的逻辑关系。事件知识关注的是通过子事件和事件参与者之间的层次关系和时间关系组织起来的关于人们生活中日常事件的结构性知识。总之，由此产生的知识将是向前迈出的关键一步，使强大的人工智能系统能够在自然语言处理和计算机视觉之间的交叉点上理解和推理非结构化的多模态信息。本研究的潜在影响包括视障人士的互动辅助系统和多模式教育界面。本项目将多模态知识提取作为一种新的研究范式，将自然语言处理中的相关方法（如信息提取、文本隐含和框架语义）与计算机视觉的最新进展联系起来。常识性知识获取的关键挑战之一是克服报告偏见，即人们不陈述显而易见的事情。因此，该项目开发了基于基于图形的集体推理的新学习算法，可以对系统地影响人们用语言、图像和视频描述世界的方式的未说出来的知识进行推理。此外，本项目还开发了视觉语义解析和事件识别的新模型，通过指定事件的各种结构组件（如参与者、对象、位置、工具、意图和目标）来概括现有的活动识别研究。学习到的知识和表征将通过包括多模态问答和基础语言理解在内的几个应用得到验证。

Everyday knowledge about the world is a necessary condition for intelligent information processing and reasoning. People can read between the lines in text and see beyond what are visible in images because of everyday functional knowledge about how the world works. The primary goal of this research is to develop learning algorithms that can automatically acquire such knowledge, centered around entities and events, from large-scale multimodal web data. Entity knowledge includes a broad range of physical and conceptual knowledge about objects and people, including their attributes, their relative differences, and logical relations among them. Event knowledge focuses on structural knowledge about everyday events in people's lives organized through hierarchical and temporal relations among sub-events and the event participants. Together, the resulting knowledge will be a critical step forward to enable robust AI systems at the intersection between natural language processing and computer vision that can understand and reason about unstructured multimodal information. The potential impact of this research includes interactive assistive systems for the visually-impaired and multimodal educational interfaces. This project investigates multimodal knowledge extraction as a new research paradigm drawing connections between relevant methods in natural language processing such as information extraction, textual entailments, and frame semantics with recent advances in computer vision. One of the critical challenges in commonsense knowledge acquisition is to overcome reporting bias, i.e., people do not state the obvious. Therefore, this project develops new learning algorithms based on a graph-based collective inference that can reason about unspoken knowledge that systematically influences the way people describe the world in language, images, and videos. In addition, this project develops new models for visual semantic parsing and event recognition, which generalize existing studies on activity recognition by specifying various structural components of events such as actors, objects, locations, tools, intents, and goals. The learned knowledge and representation will be validated through several applications including multimodal question answering and grounded language understanding.