Video Segmentation from Multiple Representations using Lifted Multicuts

视频分割是许多高级任务的重要技术，如目标检测、动作识别或三维重建。主要挑战包括：1。难以确定哪些物体是显著的。关于期望的分割细节水平的模糊性。可见图像梯度与语义对象边界之间的差异。缺乏适当的训练数据。处理大量的数据。与图像分割相比，视频数据为分割提供了许多额外的线索。视频中出现的动作似乎是物体显著性的强烈暗示。同时，运动使物体的概念更容易定义，并提高了对场景几何形状的估计。然而，目前的视频分割方法很难利用这些优势，因为它很难处理大量的数据。为了方便使用图形模型（计算成本很高），大多数最先进的视频分割方法在预先计算的逐帧超像素上构建图形。此外，基于监督学习的方法只能从当前可用的临时稀疏注释中学习。时间一致性主要通过逐帧光流来解决。在这个项目中，我们想要建立一个灵活的、概率的、基于图形的框架，用于暂时一致的视频分割。预期的框架包括各种时间和空间线索，如局部像素信息，点轨迹或物体和物体部分检测到一个目标函数，并提供在不同细节级别的所有这些表示的分割（分组）。在不同粒度级别的实体之间为给定的切割概率生成最可能的分割的动机的驱动下，该项目建立在最小成本提升多切割问题公式的基础上来生成分割。这不仅可以优化段标签分配，还可以优化段的数量，从而确定合适的对象数量。我们最重要的计划贡献是，首先，提供一个快速的最小成本提升多切口问题求解器，允许解决视频尺度的问题实例，第二，从视频中提取适当包含时间信息的边界估计，第三，生成适当优化的点轨迹，以便可以很好地表示长期运动，最后，提供了一个视频分割问题的公式，使用这些多个表示作为最小成本提升多切问题，允许联合优化。

Video segmentation is an important technique for many high-level tasks such as objectdetection, action recognition or 3D reconstruction. Key challenges include:1. The difficulty to determine which objects are salient.2. The ambiguity concerning the desired level of detail of a segmentation.3. The discrepancy between visible image gradients and semantic object boundaries.4. The lack of proper training data.5. The handling of the large amounts of data.In contrast to image segmentation, video data offers many additional cues for the segmentation. The motion present in videos appears to be a strong cue for object saliency. At the same time, motion makes the notion of objectness easier to define and gives rise to improved estimates of a scene's geometry.However, current video segmentation methods can barely make use of these advantages, because it is difficult to handle the large amount of data . In order to facilitate the use of (computationally expensive) graphical models, most state-of-the-art video segmentation methods build graphs on precomputed frame-wise superpixels. Furthermore, supervised learning based methods can only learn from the temporally sparse annotations that are currently available. Temporal consistency is mostly addressed by using frame-by-frame optical flow.In this project, we want to build a flexible, probabilistic, graph-based framework for temporally consistent video segmentation. The intended framework includes all sorts of temporal and spatial cues such as local pixel information, point trajectories or object and object part detections into one objective function and provides segmentations (groupings) for all those representations at different levels of detail.Driven by the motivation to generate the most likely segmentation for given cut probabilities between entities at different levels of granularity, this project builds upon the Minimum Cost Lifted Multicut problem formulation to generate segmentations. This enables to not only optimize over the segment label assignments but also over the number of segments and thus determine the right amount of objects.Our most important planned contributions are, first, to provide a fast Minimum Cost Lifted Multicut Problem Solver that allows to solve video-scale problem instances, second, to extract boundary estimates from video that properly include the temporal information, third, to generate properly optimized point trajectories such that long term motion can be well represented, and last, to provide a formulation of the video segmentation problem using these multiple representations as a Minimum Cost Lifted Multicut Problem, that allows for a joint optimization.