Collaborative Research: RI:Medium: Understanding Events from Streaming Video - Joint Deep and Graph Representations, Commonsense Priors, and Predictive Learning

虽然人类很容易处理视频数据并从中提取含义，但设计算法来做到这一点却极其困难。当开发出来时，这项技术有很多应用，比如建造辅助机器人，或者建造独立生活的智能空间，或者监测野生动物。视频数据捕捉事件，这是人类体验的核心内容。事件由对象/人(谁)、地点(地点)、时间(时间)、行动(什么)、活动(如何)和意图(为什么)组成。该项目开发了一种基于计算机视觉的事件理解算法，该算法以自我监督的流媒体方式运行。该算法将预测和检测新旧事件，学习建立分层事件表示，所有这些都是在随时间更新的先前知识库的背景下进行的。其意图是产生对事件的解释，超越所见，而不仅仅是认识。这项研究通过将自我监督学习过程与先验知识相结合，将该领域推向开放世界的算法，并且几乎不需要监督，从而推动了计算机视觉的前沿。此外，该项目将侧重于在大一和大二期间招收和留住本科生，关注南佛罗里达大学、佛罗里达州立大学和俄克拉荷马州立大学这三个地点未被充分代表的少数族裔学生。该方法的核心是一个混合的表征层次结构，包括连续表征和基于符号图形的表征。连续值表示是标准的向量值深度学习堆栈，它以知识库中某个对象或动作概念的嵌入向量结束。下一层次的表征由这些动词和名词的基本符号组成。当这些基本成分与知识库中的概念相关联时，它们构成了事件解释，其中包含的描述超出了图像中观察到的内容。这些象征性的层次是使用格勒南德的模式理论中的规范表示来构建的。这些表示具有灵活的图形结构主干，比其他众所周知的图形模型更具表现力。该项目的具体技术目标有四个方面。首先，它试图将基于函数的连续和基于能量的格勒南德的典型符号表示从模式理论整合为一个基于平衡传播的综合公式。第二，它将研究和开发使用和修改常识性知识库的方法。这将有助于超越封闭世界假设，这种假设隐含在当前基于注释的基于数据的深度学习方法的实践中。第三，它将在图流形上开发动态模型，这将使图结构的生成性建模成为可能，用于预测和发现新概念。第四，受人类感知实验和神经科学发现的启发，它将设计针对连续表征和符号表征的预测性自我监督学习。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

While it is easy for humans to process video data and extract meanings from it, it is extremely hard to design algorithms to do so. When developed, there are many applications of this technology, such as building assistive robotics or constructing smart spaces for independent living or monitoring wildlife. Video-data capture events, which are central to the content of human experience. Events consist of objects/people (who), location (where), time (when), actions (what), activities (how), and intent (why). This project develops a computer vision-based event understanding algorithm that operates in a self-supervised, streaming fashion. The algorithm will predict and detect old and new events, learn to build hierarchical event representations, all in the context of a prior knowledge-base that is updated over time. The intent is to generate interpretations of an event that go beyond what is seen, rather than just recognition. This research pushes the frontier of computer vision by coupling the self-supervised learning process with prior knowledge, moving the field towards open-world algorithms, and needing little or no supervision. Furthermore, this project will focus on recruitment and retention of undergraduate women students through freshman and sophomore years, with attention towards underrepresented minority students at the three sites: University of South Florida, Florida State University, and Oklahoma State University.At the core of the approach is a hybrid representational hierarchy that includes both continuous representations and symbolic graph-based representations. The continuous-valued representation is the standard, vector-valued deep learning stack that ends in an embedding vector of some object or action concept in the knowledge base. The next level of the representation consists of elementary symbolic compositions of these verbs and nouns. These elementary compositions, when associated with concepts from a knowledge-base they makeup an event interpretation, containing descriptions that go beyond what is observed in the image. These symbolic levels are built using Grenander's canonical representations from pattern theory. These representations, which have flexible graph-structured backbones, are more expressive than other well-known graphical models. The specific technical aims of the project are four-fold. First, it seeks to integrate function-based continuous with energy-based Grenander's canonical symbolic representations from pattern theory into one integrated formulation based on equilibrium propagation. Second, it will research and develop ways to use and modify commonsense knowledge bases. This will help to go beyond the closed world assumption, which is implicit in the current practice of annotated data-based deep learning approaches. Third, it will develop dynamical models on graph manifolds, which will enable generative modeling of graph structures for prediction and discovery of new concepts. Fourth, inspired by finding from human perception experiments and neuroscience, it will design predictive self-supervised learning over both continuous and symbolic representations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.