RI: Small: Grounding Probabilistic Event Logic in a Hierarchy of Video Segmentation Tubes

该项目解决了将有关事件的高级语义概念转换为低级视频数据的根本挑战。主要创新包括：(1)通过概率事件逻辑(PEL)以及相应的推理和学习算法来表示事件；(2)将视频分割成时空管的层次结构；(3)通过AND-OR语法将PEL稳健地接地到空时管中。通过跨帧跟踪候选对象边界来提取空时管，其中边界检测和跟踪都是从训练视频中学习的。PEL允许事件之间的任意、概率、时空约束，包括传统的组成规则、时间间隔之间的Allen关系以及不同事件之间的相关性。与现有的工作不同，PEL的逻辑性质允许人类，甚至非专家，很容易将他们自己的知识注入系统中。PEL进行联合的整体推理，以找到所有事件的全局最佳解析，其基础是原始事件的AND-OR语法。AND-OR语法使用视频管的健壮图形匹配来处理低级视觉处理中的不确定性。为了进行评估，编制了美式足球和一栋建筑-S中庭的两个视频数据集，包括完全注释的事件标签、对象轨迹和时空分割。为研究生和本科生提供培训，包括来自任职人数不足群体的学生。预计该项目将：(A)提高通常只侧重于视频分类的最新水平；(B)公布两个数据集；(C)就相关专题编写讲习班/教程；(D)在影响最大的期刊/会议上发表出版物。

The project addresses the fundamental challenge of grounding high-level semantic concepts about events into low-level video data. The key innovations include: (1) Representing events via probabilistic event logic (PEL) along with corresponding inference and learning algorithms, (2) Video segmentation into a hierarchy of space-time tubes, and (3) Robustly grounding PEL into space-time tubes via AND-OR grammars. Space-time tubes are extracted by tracking candidate object boundaries across frames, where both boundary detection and tracking are learned from training videos. PEL allows for arbitrary, probabilistic, spatiotemporal constraints among events, including the traditional compositional rules, Allen relations between time intervals, and correlations among different events. Unlike existing work, the logical nature of PEL allows humans, even non-experts, to easily inject their own knowledge into the system. PEL conducts joint, holistic inference to find the globally best parse over all events, which is grounded in an AND-OR grammar of primitive events. The AND-OR grammar uses robust graph matching of video tubes for handling uncertainty in low-level visual processing. For evaluation, two video datasets of American football and a building?s atrium are compiled, with fully annotated event labels, object tracks, and spatiotemporal segmentations. Training is provided for graduate and undergraduate students, including those from under-represented groups. The project is expected to: (a) advance the state of the art which typically focuses only on video classification; (b) make the two datasets public; (c) generate workshops/tutorials on the related topics; and (d) produce publications in the highest-impact journals/conferences.