Towards Accurate Detection, Segmentation, and Tracking of Objects and other Scene Attributes

能够识别、分割和跟踪视频中的各个对象是有益的，包括刚性和铰接对象（例如，人、动物、汽车等），背景元素（天空、云、树等），标记（即，专门设计为向相机显示高对比度的对象）和特征点（固定的高对比度像素，通常是角，可以在多个帧上轻松跟踪并用作参考点）。除了能够准确地将每个像素分配给特定对象之外，还可以确定它们的位置（如果可能的话，在真正的3D空间中），并通过多个视频帧跟踪它们，即使它们的外观发生变化，或者它们在场景中移动时被其他对象部分/完全遮挡。该技术用于一系列应用中，包括：视频压缩、来自单个和立体摄像机的深度估计（例如机器人）、多摄像机拼接（以补偿视差）、视觉效果的匹配移动、自动摄像机校准系统、基于标记和无标记的运动捕捉以及自主车辆视觉系统等等。当前系统的局限性在于难以提取区分两个对象的精确轮廓（尤其是当一个像素可以表示两者的混合时）。其原因包括：缺乏对比度、颜色变化、纹理或照明;相机视角和对象方向频繁变化;运动模糊（由于快速移动的场景和/或相机，或低光-意味着较慢的快门速度）;灯光、阴影和反射变化（在静态和变化条件下），以及摄像机内固有的各种其他较小的伪像（包括传感器噪声以及简单地大多数传感器是拜耳瓦片类型的事实，因此帧是通过去马赛克构造的）。此外，大多数应用需要实时处理，以便其他系统可以对信息做出反应（例如自动驾驶汽车，周边安全）。因此，识别单个物体，确定它们的形状和轮廓到亚像素精度，并在一系列帧中准确地跟踪它们对于许多关键任务应用至关重要，特别是在各种不利条件下（例如下雪或下雨），而无需依赖额外的昂贵技术（如激光雷达/雷达）在这项研究计划中，我们提出了一系列新的和互补的研究方向，这些研究方向基于传统视频处理技术和深度学习方法的结合，以提高现有技术的准确性，并处理在应用领域中可以预期的更加变化和复杂的条件，特别是在条件不能被控制但可以依赖和信任它们以产生精确的实时结果的情况下。

There is benefit in being able to identify, segment, and track individual objects in video, including rigid and articulated objects (e.g. humans, animals, cars, etc.), background elements (sky, clouds, trees, etc.), markers (i.e. objects specifically design to show as high-contrast to a camera), and feature points (fixed, high-contrast pixels, usually corners, that can be easily tracked over multiple frames and used as reference points). Apart from being able to accurately assign each pixel to a specific object, it is useful to determine where they are placed (in true 3D space if possible), and keep track of them through multiple video frames even as their appearance changes or they are partially/fully occluded by other objects as they move through the scene. This technology is used in a range of applications, including: video compression, depth estimation from single and stereo cameras (e.g. robotics), multi-camera stitching (to compensate for parallax), match-moving for visual effects, automatic camera calibration systems, marker-based and marker-less motion capture, and autonomous vehicle vision systems to name a few. The limitation of current systems is that it is difficult to extract the exact outline that distinguishes between two objects (especially as one pixel can represent a blending of both). The reasons for this include: lack of contrast, colour variation, texture, or lighting; frequent shifts in camera perspective and object orientation; motion blur (due to a fast moving scene and/or camera, or low light - meaning slower shutter speeds); light, shadow, and reflection variations (both under static and changing conditions), and various other smaller artifacts that are inherent within the camera (including sensor noise and simply the fact most sensors are Bayer-tile type and thus frames are constructed through demosaicing). In addition, most applications require real-time processing so that other systems can react to the information (e.g. autonomous vehicles, perimeter security). Therefore, identifying individual objects, determining their shape and outlines to sub-pixel accuracy and accurately tracking them in a sequence of frames is crucial for many mission-critical applications, especially under varying adverse conditions (e.g. snow or rain), without the need to rely on additional costly technology (such as LIDAR/RADAR) In this research program, we propose a range of new and complementary research directions based on a combination of traditional video processing techniques and deep learning methods that can be used to improve the accuracy of the current state of the art and deal with the more varied and complex conditions that can be expected in the application areas especially where the conditions cannot be controlled, but where they can be relied upon and trusted to produce precise real-time results.