Spatial and Temporal Filtering of Depth Data for Telepresence

在过去的几年中，人们对遥现通信系统的兴趣越来越大，遥现通信系统产生了存在于与真实位置不同的地方的印象。这一领域的一个主要挑战是在发送站点将获取的图像处理成真实的场景的高质量3D表示。高质量的方法通常需要密集的离线处理。而且，在真实的时间中工作的方法产生低等级的3D表示。在最近的系统中，这是由诸如Microsoft Kinect之类的低成本深度传感器的应用引起的，其在真实的时间内提供3D表示，但仍然表现出相当数量的干扰伪像。为了理解它们强烈的时间分量，我们将从实验性地开发一个常见深度传感器失真的统计模型开始。与此领域现有的工作相比，我们还将考虑时间方面的问题。在分析所获得的数据的结果的指导下，我们将开发一种新的实时时空滤波器，以同时稳定空间和时间域中的失真深度数据。因此，我们提出了一种新的深度离群点检测方法，深度相机的运动估计和3D滤波的组合。我们的想法是平滑每个深度像素的三维空间和时间邻域的基础上。为了识别深度帧流中的时间相关邻居，我们将估计场景对象的运动以追溯每个深度像素的历史。由于深度图像对于这项任务来说太不稳定，我们建议对彩色图像进行运动估计，通常由当前的深度传感器提供。利用彩色相机和深度相机之间固定的紧密空间关系，将估计结果传递到深度图像中，在编译一帧中所有深度像素的时空3D邻域后，使用6D线性回归插入鲁棒的离群点检测和去除步骤。在这里，一个必要的研究将投入到如何实现最小中位数的平方方法，这一方面，是适合解决的任务，但另一方面，是很难做到在真实的时间的问题。在清除异常值之后，过滤3D深度邻域将去除固有的高斯噪声。我们的新方法将被集成到由RGB-D相机阵列组成的远程呈现原型系统中。在这里，我们计划通过将我们以前的工作扩展到动态3D表示来交叉验证来自多个相机的校正深度数据。为了评估我们提出的方法，测试数据将与地面实况一起生成，这些地面实况是从预定义的场景或添加了硬件符合噪声的人工图像中获得的。

In the past years, there has been a growing interest in telepresence communication systems, which create the impression of being present at a place different from the true location. A major challenge in this area is to process the acquired imagery at the sender site into a high-quality 3D representation of the scene in real time. High quality approaches usually require intensive offline processing. Then again, methods that work in real time produce 3D representations at a low grade. In recent systems this is caused by the application of low-cost depth sensors, such as Microsoft Kinect, which deliver 3D representations in real time but still exhibit considerable amounts of disturbing artifacts.The flickering nature of artifacts is often not considered. To understand their strong temporal component, we will start with experimentally developing a statistical model for the distortions of common depth sensors. In contrast to existing work in this field, we will also consider the temporal aspects of the matter.Guided by the results of analyzing the gained data, we will develop a new real-time spatio-temporal filter to simultaneously stabilize distorted depth data in the spatial and temporal domains. Therefore, we suggest a composition of a novel depth outlier detection method, motion estimation for depth cameras and 3D-filtering.Our idea is to smooth every depth pixel based on its 3D spatial and temporal neighborhood. In order to identify temporally related neighbors in the stream of depth frames, we will estimate the motion of scene objects to trace back the history of every depth pixel. As depth images are too unstable for this task, we suggest to rather perform motion estimation on the color images, usually delivered alongside by current depth sensors. By the fixed close spatial relation between color and depth camera we can transfer the estimation results to the depth images.After compiling the spatio-temporal 3D neighborhood of all depth pixels in a frame, we will insert a robust outlier detection and removal step using 6D linear regression. Here, an essential amount of research will be invested into the question on how to implement a least median of squares approach, which, on the one hand, is suitable to solve the task but, on the other hand, is difficult to do in real time. After cleaning away the outliers, filtering the 3D depth neighborhood will remove the inherent Gaussian noise.Our new approach will be integrated into a telepresence prototype system comprising an array of RGB-D cameras. Here, we plan to cross-validate the corrected depth data from multiple cameras by extending our previous work towards dynamic 3D representations. For the evaluation of our proposed method, testing data will be generated alongside with ground truth either obtained from predefined scenery or from artificial imagery with added hardware-conform noise.