Spatio-Temporal Hypercolumns for Instance-based Semantic Segmentation in Video

视频分割是计算机视觉中最具挑战性的开放性问题之一。尽管文献中提出了多种方法来解决这一任务，但在现实的无约束视频中，最先进的算法仍远未达到人类水平的性能。在这项工作中，我们提出了一个为期两年的研究计划，重点研究视频分割和目标识别之间的相互作用，从而引入特定类别的信息，以改进视频分割过程。作为我们方法的起点，我们将通过考虑光流估计，将最先进的静态通用分割和语义分割算法推广到时空域。在我们为期两年的研究项目的第一年，我们将寻求我们最近的工作卷积面向边界（COB）和FlowNet的有效结合，以建立一个涉及局部和空间信息以及时间一致性的时空视频分割算法。一旦我们提取了一致的时空视频分割，我们将通过轨迹运动亲和力沿帧传播像素标签，并为物体和表面建立一个我们称之为卷积时间管（CTT）的时空表示。在第二年，我们将通过在CTT上实例化一个时空超列框架来扩展我们之前在Hypercolumns[3]上的工作，以便在保持时间一致性的同时，根据对象和表面的语义特征来改进其空间支持。这个为期两年的研究项目的最终目标是在意识到语义和单个对象实例的同时，根据时空区域稳定地呈现视频。我们的研究计划的实现有望在当前基准的视频分割中弥合人类和计算机性能之间的差距。这些结果将推动在场景和物体结构恢复、3D重建、视频理解、动作和物体识别以及许多其他应用方面的进一步研究。该项目旨在加强德国和哥伦比亚之间的科学交流，并将在两国研究人员的密切合作下进行。

Video segmentation is one of the most challenging open problems in computer vision. Although multiple approaches have been proposed in the literature to address this task, state-of-the-art algorithms are still far from reaching human-level performance in realistic unconstrained videos. In this work, we propose a two year research program that focuses on studying the interaction between video segmentation and object recognition, introducing thus category-specific information in order to improve the video segmentation process. As starting point of our approach, we will generalize to the spatio-temporal domain state-of-the-art algorithms for static generic segmentation and semantic segmentation, by taking into account optical flow estimation. During the first year of our two year research project, we will seek for an effective combination of our recent works Convolutional Oriented Boundaries (COB) and FlowNet, in order to build a spatio-temporal video segmentation algorithm that involves local and spatial information as well as temporal consistency. Once we have extracted a consistent spatio-temporal video segmentation, we will propagate the pixel labels along frames through trajectory motion affinities and build a spatio-temporal representations for the objects and surfaces which we call Convolutional Temporal Tubes (CTT). During the second year, we will extend our previous work on Hypercolumns [3] by instantiating a spatiotemporalhypercolumn framework on the CTT, in order to refine the spatial support of objects and surfaces given their semantic characteristics while preserving temporal consistency. This representation of a video in terms of spatio-temporal regions that are stable over time while being aware of semantics and of individual instances of objects is the final objective for this two year research project. The realization of our research programme is expected to bridge the gap between human and computer performance in video segmentation for the current benchmarks. These results will enable further research in scene and object structure recovery, 3D reconstruction, video understanding, actionand object recognition, among many other applications. This project seeks to strengthen scientific exchanges between Germany and Colombia, and will be conducted in close collaboration by researchers in both countries.