High Quality 3D Geometry and Appearance Reconstruction of Non-Rigidly Deforming Objects using Low-Cost RGB-D Cameras

捕捉和重建高质量的3D几何体和非刚性变形对象的外观，如人体动作的动力学，对于许多应用程序来说都是必不可少的，包括创意行业的电影和游戏制作、虚拟现实(VR)视频会议、用于医疗监测和运动分析的分析人类行为等。尽管付出了巨大的努力，但这仍然是一个具有挑战性的问题，特别是当涉及大规模变形时：自我遮挡、细微的几何形状和外观变化(如皮肤和衣服的皱纹)都是造成困难的原因。这项研究旨在通过研究新的数据驱动技术来解决根本挑战，从而推动最新技术的发展。近年来，低成本的RGB深度相机变得更有能力，并将在研究中使用，使这项技术得到广泛应用。我们将开发一种新的几何和外观联合表示和分析技术，以有效地编码非刚性变形过程中的几何和外观变化。看似合理的变形和外观变化通常形成嵌入在这个关节空间中的低维流形。为了解决扫描数据中的噪声和不完整性问题，将利用机器学习技术，如流形学习。这将有效地利用以前任何扫描的信息来填补空白并提高重建质量。为了使研究可行，该项目建立在主管在形状变形(表示和形状空间分析)、非刚性配准、数据驱动重建和稀疏模型方面的现有工作的基础上，所有这些都最近在顶级期刊上发表了文章。该项目是与英国皇家学会目前与中国领先研究机构的两个国际合作项目的相关和补充。这项研究和博士生将从这些合作中受益匪浅。

Capturing and reconstruction of high quality 3D geometry and the appearance of non-rigidly deforming objects, such as the dynamics of human actions, is essential for many applications, including movie and game production in the creative industries, Virtual Reality (VR) videoconferencing, analysing human behaviour for healthcare monitoring and sports analysis etc. Despite great effort, it is still a challenging problem, especially when large-scale deformations are involved: self-occlusion, subtle geometry and appearance change (e.g. wrinkles of skin and clothing) all contribute to the difficulty. This research aims to advance the state of the art by investigating novel data-driven techniques to address fundamental challenges. Low-cost RGB-Depth cameras have become more capable in recent years and will be used in the research to make the techniques widely useful. We will develop a new joint representation and analysis technique for both geometry and appearance, to effectively encode geometric and appearance change during non-rigid deformation. The plausible deformation and change in appearance typically form a low dimensional manifold embedded in this joint space. To address the issues of noise and incompleteness in the scanned data, machine learning techniques such as manifold learning will be exploited. This will effectively utilise information from any previous scans to fill the gaps and improve the quality of reconstruction. An optimisation framework will also be developed incorporating knowledge from the manifold as well as sparse priors.To make the research feasible, the project is built on top of the supervisors' existing work on shape deformation (representation and shape space analysis), non-rigid registration, data-driven reconstruction and sparse models, all of which have recent publications in top journals. The project is related and complementary to two current Royal Society international collaborative projects with leading research institutions in China. The research and the PhD student will benefit greatly from these collaborations.