Spatio-Temporal Hypercolumns for Instance-based Semantic Segmentation in Video

视频分割是计算机视觉中最具挑战性的开放性问题之一。虽然在文献中已经提出了多种方法来解决这个任务，最先进的算法仍然远远没有达到人类水平的性能，在现实的无约束的视频。在这项工作中，我们提出了一个为期两年的研究计划，重点研究视频分割和对象识别之间的相互作用，引入特定于类别的信息，以改进视频分割过程。作为我们的方法的出发点，我们将推广到时空域的静态通用分割和语义分割的最先进的算法，考虑到光流估计。在我们为期两年的研究项目的第一年，我们将寻求我们最近的工作卷积导向边界（COB）和FlowNet的有效结合，以建立一个时空视频分割算法，涉及本地和空间信息以及时间一致性。一旦我们提取了一致的时空视频分割，我们将通过轨迹运动亲和度沿着沿着帧传播像素标签，并为对象和表面构建时空表示，我们称之为卷积时间管（CTT）。在第二年，我们将通过在CTT上实例化一个时空超列框架来扩展我们之前在超列[3]上的工作，以便在保持时间一致性的同时，根据对象和表面的语义特征来细化它们的空间支持。这种在时空区域方面的视频表示，随着时间的推移是稳定的，同时意识到语义和对象的个体实例，是这个为期两年的研究项目的最终目标。我们的研究计划的实现有望弥合人类和计算机之间的差距差距在视频分割为目前的基准。这些结果将使进一步的研究在场景和对象结构恢复，三维重建，视频理解，actionand对象识别，以及许多其他应用。该项目旨在加强德国和哥伦比亚之间的科学交流，并将由两国研究人员密切合作进行。

Video segmentation is one of the most challenging open problems in computer vision. Although multiple approaches have been proposed in the literature to address this task, state-of-the-art algorithms are still far from reaching human-level performance in realistic unconstrained videos. In this work, we propose a two year research program that focuses on studying the interaction between video segmentation and object recognition, introducing thus category-specific information in order to improve the video segmentation process. As starting point of our approach, we will generalize to the spatio-temporal domain state-of-the-art algorithms for static generic segmentation and semantic segmentation, by taking into account optical flow estimation. During the first year of our two year research project, we will seek for an effective combination of our recent works Convolutional Oriented Boundaries (COB) and FlowNet, in order to build a spatio-temporal video segmentation algorithm that involves local and spatial information as well as temporal consistency. Once we have extracted a consistent spatio-temporal video segmentation, we will propagate the pixel labels along frames through trajectory motion affinities and build a spatio-temporal representations for the objects and surfaces which we call Convolutional Temporal Tubes (CTT). During the second year, we will extend our previous work on Hypercolumns [3] by instantiating a spatiotemporalhypercolumn framework on the CTT, in order to refine the spatial support of objects and surfaces given their semantic characteristics while preserving temporal consistency. This representation of a video in terms of spatio-temporal regions that are stable over time while being aware of semantics and of individual instances of objects is the final objective for this two year research project. The realization of our research programme is expected to bridge the gap between human and computer performance in video segmentation for the current benchmarks. These results will enable further research in scene and object structure recovery, 3D reconstruction, video understanding, actionand object recognition, among many other applications. This project seeks to strengthen scientific exchanges between Germany and Colombia, and will be conducted in close collaboration by researchers in both countries.