RI: Small: Grounding Probabilistic Event Logic in a Hierarchy of Video Segmentation Tubes

该项目解决了将事件的高级语义概念根植到低级视频数据中的基本挑战。关键创新包括：(1)通过概率事件逻辑（PEL）以及相应的推理和学习算法来表示事件；(2)将视频分割成时空管道的层次结构；(3)通过与或语法将PEL稳健地建立在时空管道中。通过跨帧跟踪候选目标边界提取时空管，其中边界检测和跟踪都是从训练视频中学习的。PEL允许事件之间的任意、概率、时空约束，包括传统的组合规则、时间间隔之间的Allen关系以及不同事件之间的相关性。与现有的工作不同，PEL的逻辑特性允许人类，甚至是非专家，轻松地将自己的知识注入系统。PEL以原始事件的AND-OR语法为基础，进行联合的、整体的推理，以找到所有事件的全局最佳解析。与或语法使用视频管的鲁棒图匹配来处理低级视觉处理中的不确定性。为了评估，两组橄榄球和一栋建筑的视频数据？S心房编译，具有完全标注的事件标签、对象轨迹和时空分割。为研究生和本科生提供培训，包括来自代表性不足群体的学生。预计该项目将：(a)提高通常只关注视频分类的技术水平；(b)公开两个数据集；(c)举办有关主题的工作坊/教程；(d)在最具影响力的期刊/会议上发表文章。

The project addresses the fundamental challenge of grounding high-level semantic concepts about events into low-level video data. The key innovations include: (1) Representing events via probabilistic event logic (PEL) along with corresponding inference and learning algorithms, (2) Video segmentation into a hierarchy of space-time tubes, and (3) Robustly grounding PEL into space-time tubes via AND-OR grammars. Space-time tubes are extracted by tracking candidate object boundaries across frames, where both boundary detection and tracking are learned from training videos. PEL allows for arbitrary, probabilistic, spatiotemporal constraints among events, including the traditional compositional rules, Allen relations between time intervals, and correlations among different events. Unlike existing work, the logical nature of PEL allows humans, even non-experts, to easily inject their own knowledge into the system. PEL conducts joint, holistic inference to find the globally best parse over all events, which is grounded in an AND-OR grammar of primitive events. The AND-OR grammar uses robust graph matching of video tubes for handling uncertainty in low-level visual processing. For evaluation, two video datasets of American football and a building?s atrium are compiled, with fully annotated event labels, object tracks, and spatiotemporal segmentations. Training is provided for graduate and undergraduate students, including those from under-represented groups. The project is expected to: (a) advance the state of the art which typically focuses only on video classification; (b) make the two datasets public; (c) generate workshops/tutorials on the related topics; and (d) produce publications in the highest-impact journals/conferences.