Towards Accurate Detection, Segmentation, and Tracking of Objects and other Scene Attributes

能够识别、分割和跟踪视频中的单个对象是有好处的，包括刚性和铰接对象（例如人类、动物、汽车等）、背景元素（天空、云、树木等）、标记（即专门设计用于显示与相机高对比度的对象）和特征点（固定的、高对比度的像素，通常是角落，可以在多个帧上轻松跟踪并用作参考点）。除了能够准确地将每个像素分配给特定对象之外，确定它们的位置（如果可能的话，在真正的3D空间中）并通过多个视频帧跟踪它们也是有用的，即使它们的外观发生了变化，或者它们在场景中移动时被其他对象部分/完全遮挡。该技术用于一系列应用，包括：视频压缩、单摄像机和立体摄像机（例如机器人）的深度估计、多摄像机拼接（以补偿视差）、视觉效果的匹配移动、自动摄像机校准系统、基于标记和无标记的运动捕捉以及自动车辆视觉系统等等。当前系统的限制是很难提取出区分两个物体的确切轮廓（特别是当一个像素可以代表两者的混合时）。造成这种情况的原因包括：缺乏对比度、色彩变化、纹理或照明；相机视角和物体方向的频繁变化；动态模糊（由于快速移动的场景和/或相机，或低光-意味着较慢的快门速度）；光线，阴影和反射变化（在静态和变化条件下），以及相机内部固有的各种其他较小的工件（包括传感器噪声和大多数传感器是拜耳瓦类型的事实，因此通过去马赛克构建帧）。此外，大多数应用程序需要实时处理，以便其他系统可以对信息做出反应（例如自动驾驶汽车，周边安全）。因此，识别单个物体，确定其形状和轮廓到亚像素精度，并在一系列帧中准确跟踪它们对于许多关键任务应用至关重要，特别是在不同的不利条件下（例如雪或雨），而不需要依赖额外的昂贵技术（如激光雷达/雷达）。基于传统视频处理技术和深度学习方法的结合，我们提出了一系列新的和互补的研究方向，可用于提高当前技术水平的准确性，并处理应用领域中可能出现的更多样化和更复杂的条件，特别是在条件无法控制的情况下，但在这些情况下，它们可以被依赖和信任以产生精确的实时结果。

There is benefit in being able to identify, segment, and track individual objects in video, including rigid and articulated objects (e.g. humans, animals, cars, etc.), background elements (sky, clouds, trees, etc.), markers (i.e. objects specifically design to show as high-contrast to a camera), and feature points (fixed, high-contrast pixels, usually corners, that can be easily tracked over multiple frames and used as reference points). Apart from being able to accurately assign each pixel to a specific object, it is useful to determine where they are placed (in true 3D space if possible), and keep track of them through multiple video frames even as their appearance changes or they are partially/fully occluded by other objects as they move through the scene. This technology is used in a range of applications, including: video compression, depth estimation from single and stereo cameras (e.g. robotics), multi-camera stitching (to compensate for parallax), match-moving for visual effects, automatic camera calibration systems, marker-based and marker-less motion capture, and autonomous vehicle vision systems to name a few. The limitation of current systems is that it is difficult to extract the exact outline that distinguishes between two objects (especially as one pixel can represent a blending of both). The reasons for this include: lack of contrast, colour variation, texture, or lighting; frequent shifts in camera perspective and object orientation; motion blur (due to a fast moving scene and/or camera, or low light - meaning slower shutter speeds); light, shadow, and reflection variations (both under static and changing conditions), and various other smaller artifacts that are inherent within the camera (including sensor noise and simply the fact most sensors are Bayer-tile type and thus frames are constructed through demosaicing). In addition, most applications require real-time processing so that other systems can react to the information (e.g. autonomous vehicles, perimeter security). Therefore, identifying individual objects, determining their shape and outlines to sub-pixel accuracy and accurately tracking them in a sequence of frames is crucial for many mission-critical applications, especially under varying adverse conditions (e.g. snow or rain), without the need to rely on additional costly technology (such as LIDAR/RADAR) In this research program, we propose a range of new and complementary research directions based on a combination of traditional video processing techniques and deep learning methods that can be used to improve the accuracy of the current state of the art and deal with the more varied and complex conditions that can be expected in the application areas especially where the conditions cannot be controlled, but where they can be relied upon and trusted to produce precise real-time results.