Reconstruction of Complex Deformations in 3D Scenes from Color and Depth Images

虽然在过去的二十年中，刚性环境的三维重建已经得到了非常深入的研究，但变形场景的分析仍然是一个激烈的研究课题。大多数可变形对象的重建算法利用输入图像序列中的特征点跟踪，然而，特征跟踪需要图像中足够数量的纹理或结构以及这些特征的一定程度的无歧义性。这不仅适用于彩色图像，也适用于深度图像，因为基于深度的重建需要结构信息以获得稳定的跟踪结果。由于基于特征的跟踪方法的输入数据已经是解释的结果，这类算法固有的一些缺点：1。将复杂的图像数据简化为一组特征坐标确实丢弃了许多可以用来解决重建问题的潜在信息。优化不直接使用测量数据，这是不准确的，并且容易导致图像噪声的传播。特征对应中异常值的存在要求算法识别错误信息并予以丢弃。这导致整体优化问题的收敛性较弱。这个问题可以通过使用直接操作输入数据的方法来规避：基于“合成分析”（AbS）的方法生成跟踪对象的3D模型，并用它来合成虚拟相机图像。然后利用合成图像与实际输入图像之间的差异来分析目标参数。该系统的缺点是需要一个广泛的全局优化过程，因为问题不能在其目标参数中线性化。然而，AbS利用了完整的图像数据。本研究项目的主要目标是结合现有AbS方法和基于特征的算法的优点，同时保持直接优化的有用性质。这将通过使用有关特征移动的信息来为直接优化过程提供“指导”而不影响直接优化目标来实现。除此之外，还将构建自适应变形模型，以保证通用性，同时降低计算优化成本。

While the 3D reconstruction of rigid environments has been investigated very thoroughly in the last two decades, the analysis of deforming scenes is still subject to intense research. Most reconstruction algorithms for deformable objects utilize feature point tracking in a sequence of input images, however, feature tracking requires a sufficient amount of texture or structure in the image as well as a certain level of unambiguity of these features. This does not only apply to color images but also to depth images, since depth based reconstruction requires structural information for a stable tracking result.Since the input data of feature based tracking methods is already the result of an interpretation, a number of disadvantages are inherent to this class of algorithms:1. The reduction of complex image data to a set of feature coordinates does discard a lot of information that can potentially be used to solve the reconstruction problem.2. Optimization does not employ the measurement data directly, which is inaccurate and entails error prone propagation of image noise.3. The existence of outliers in the feature correspondences requires the algorithm to identify wrong information and discard it. This leads to a weaker convergence behavior of the overall optimization problem.This issues can be circumvented by using methods operating directly on the input data: 'Analysis by Synthesis' (AbS) based methods generate a 3D model of the tracked object and use it to synthesize a virtual camera image. The difference between the synthesized images and the real input images is then used to analyze the object parameters. A disadvantage of this system is the need for an extensive global optimization process, since the problem can not be linearized in its object parameters. However, AbS makes use of the complete image data available.The main goal of this research project is to combine the advantages of existing AbS methods and feature based algorithms while maintaining the useful properties of direct optimization. This is going to be achieved by using information about feature movement to provide 'guidance' to the direct optimization process without contaminating the direct optimization goal. In addition to this, an adaptive deformation model will be constructed to ensure versatility while reducing the computational optimization costs.