Video Segmentation from Multiple Representations using Lifted Multicuts

视频分割是目标检测、动作识别或三维重建等高层次任务的重要技术。主要挑战包括：1.难以确定哪些对象是显著的。关于分割的期望细节级别的模棱两可。可视图像梯度和语义对象边界之间的差异。缺乏适当的训练数据。与图像分割相比，视频数据为分割提供了许多额外的线索。视频中出现的运动似乎是物体显著程度的强烈暗示。同时，运动使得客观性的概念更容易定义，并提高了对场景几何形状的估计。然而，现有的视频分割方法很难利用这些优势，因为它很难处理大量的数据。为了便于使用(计算昂贵的)图形模型，大多数最先进的视频分割方法在预计算的帧方式超像素上构建图形。此外，基于监督学习的方法只能从当前可用的时间稀疏标注中学习。时间一致性主要是通过使用逐帧光流来解决的。在这个项目中，我们想要建立一个灵活的、基于概率的、基于图的时间一致性视频分割框架。该框架将各种时间和空间线索，如局部像素信息、点轨迹或对象和对象部分检测等包含在一个目标函数中，并为所有这些表示提供不同细节级别的分割(分组)，在不同粒度的实体之间为给定割概率生成最可能分割的动机，该项目建立在最小成本提升多点问题公式的基础上来生成分割。我们最重要的计划贡献是，第一，提供快速的最小成本提升多点问题求解器，允许解决视频规模的问题实例；第二，从视频中提取适当包括时间信息的边界估计；第三，生成适当优化的点轨迹，以便能够很好地表示长期运动；以及最后，使用这些多重表示作为允许联合优化的最小成本提升多点问题来提供视频分割问题的公式。

Video segmentation is an important technique for many high-level tasks such as objectdetection, action recognition or 3D reconstruction. Key challenges include:1. The difficulty to determine which objects are salient.2. The ambiguity concerning the desired level of detail of a segmentation.3. The discrepancy between visible image gradients and semantic object boundaries.4. The lack of proper training data.5. The handling of the large amounts of data.In contrast to image segmentation, video data offers many additional cues for the segmentation. The motion present in videos appears to be a strong cue for object saliency. At the same time, motion makes the notion of objectness easier to define and gives rise to improved estimates of a scene's geometry.However, current video segmentation methods can barely make use of these advantages, because it is difficult to handle the large amount of data . In order to facilitate the use of (computationally expensive) graphical models, most state-of-the-art video segmentation methods build graphs on precomputed frame-wise superpixels. Furthermore, supervised learning based methods can only learn from the temporally sparse annotations that are currently available. Temporal consistency is mostly addressed by using frame-by-frame optical flow.In this project, we want to build a flexible, probabilistic, graph-based framework for temporally consistent video segmentation. The intended framework includes all sorts of temporal and spatial cues such as local pixel information, point trajectories or object and object part detections into one objective function and provides segmentations (groupings) for all those representations at different levels of detail.Driven by the motivation to generate the most likely segmentation for given cut probabilities between entities at different levels of granularity, this project builds upon the Minimum Cost Lifted Multicut problem formulation to generate segmentations. This enables to not only optimize over the segment label assignments but also over the number of segments and thus determine the right amount of objects.Our most important planned contributions are, first, to provide a fast Minimum Cost Lifted Multicut Problem Solver that allows to solve video-scale problem instances, second, to extract boundary estimates from video that properly include the temporal information, third, to generate properly optimized point trajectories such that long term motion can be well represented, and last, to provide a formulation of the video segmentation problem using these multiple representations as a Minimum Cost Lifted Multicut Problem, that allows for a joint optimization.