Towards Accurate Detection, Segmentation, and Tracking of Objects and other Scene Attributes

能够识别、分割和跟踪视频中的各个对象是有益的，包括刚性和铰接式对象(例如，人、动物、汽车等)、背景元素(天空、云、树等)、标记(即，专门设计成对相机显示为高对比度的对象)和特征点(固定的、高对比度的像素，通常是角，可以容易地在多个帧上跟踪并用作参考点)。除了能够准确地将每个像素指定给特定对象外，还可以确定它们的放置位置(如果可能，在真正的3D空间中)，并通过多个视频帧跟踪它们，即使它们的外观发生变化，或者它们在场景中移动时被其他对象部分/完全遮挡。这项技术用于一系列应用中，包括：视频压缩、单摄像机和立体摄像机(例如机器人)的深度估计、多摄像机拼接(以补偿视差)、视觉效果的匹配移动、自动摄像机校准系统、基于标记和无标记的运动捕获以及自主车辆视觉系统等等。当前系统的局限性是很难提取区分两个对象的准确轮廓(特别是当一个像素可以表示两者的混合时)。其原因包括：缺乏对比度、颜色变化、纹理或照明；相机视角和对象方向的频繁变化；运动模糊(由于快速移动的场景和/或相机，或弱光-意味着较慢的快门速度)；光、阴影和反射变化(在静态和变化的条件下)，以及相机固有的各种其他较小的伪像(包括传感器噪声和大多数传感器是拜尔瓦式的，因此帧是通过去马赛克构建的)。此外，大多数应用程序需要实时处理，以便其他系统可以对信息做出反应(例如自动驾驶车辆、周边安全)。因此，识别单个物体、将它们的形状和轮廓确定为亚像素精度并在帧序列中准确地跟踪它们对于许多任务关键应用是至关重要的，特别是在各种不利条件(例如，雪或雨)下，而不需要在该研究计划中依赖额外昂贵的技术(例如激光雷达/雷达)，我们基于传统视频处理技术和深度学习方法的组合，提出了一系列新的和互补的研究方向，这些方法可以用来提高当前技术水平的准确性，并处理应用领域中预期的更多样和更复杂的条件，特别是在条件无法控制的情况下，但在可以依赖和信任的情况下，可以产生精确的实时结果。

There is benefit in being able to identify, segment, and track individual objects in video, including rigid and articulated objects (e.g. humans, animals, cars, etc.), background elements (sky, clouds, trees, etc.), markers (i.e. objects specifically design to show as high-contrast to a camera), and feature points (fixed, high-contrast pixels, usually corners, that can be easily tracked over multiple frames and used as reference points). Apart from being able to accurately assign each pixel to a specific object, it is useful to determine where they are placed (in true 3D space if possible), and keep track of them through multiple video frames even as their appearance changes or they are partially/fully occluded by other objects as they move through the scene. This technology is used in a range of applications, including: video compression, depth estimation from single and stereo cameras (e.g. robotics), multi-camera stitching (to compensate for parallax), match-moving for visual effects, automatic camera calibration systems, marker-based and marker-less motion capture, and autonomous vehicle vision systems to name a few. The limitation of current systems is that it is difficult to extract the exact outline that distinguishes between two objects (especially as one pixel can represent a blending of both). The reasons for this include: lack of contrast, colour variation, texture, or lighting; frequent shifts in camera perspective and object orientation; motion blur (due to a fast moving scene and/or camera, or low light - meaning slower shutter speeds); light, shadow, and reflection variations (both under static and changing conditions), and various other smaller artifacts that are inherent within the camera (including sensor noise and simply the fact most sensors are Bayer-tile type and thus frames are constructed through demosaicing). In addition, most applications require real-time processing so that other systems can react to the information (e.g. autonomous vehicles, perimeter security). Therefore, identifying individual objects, determining their shape and outlines to sub-pixel accuracy and accurately tracking them in a sequence of frames is crucial for many mission-critical applications, especially under varying adverse conditions (e.g. snow or rain), without the need to rely on additional costly technology (such as LIDAR/RADAR) In this research program, we propose a range of new and complementary research directions based on a combination of traditional video processing techniques and deep learning methods that can be used to improve the accuracy of the current state of the art and deal with the more varied and complex conditions that can be expected in the application areas especially where the conditions cannot be controlled, but where they can be relied upon and trusted to produce precise real-time results.