Data Driven Human Motion Synthesis and 3D Reconstruction

我的研究项目主要是三维（3D）图形。首要目标是有效地合成数字对象和世界（虚拟或增强现实- VR/AR），并轻松与它们交互。目前的重点是使用感测数据和图形处理（GPU）来开发数据驱动方法，应用于游戏，娱乐，工程和虚拟仿真。虽然早期的工作主要是使用手工制作特征的算法方法，但本文的主要思想是开发应用机器/深度学习和机器推理的新方法来解决3D重建和人体运动中的3D图形问题。提出的研究有两个核心研究目标和一个技术目标。首先是从运动数据中创造自然的人类动作序列（用于游戏，电影和VR/AR）。第二种是从感知的视觉数据中自动重建3D物体和更大场景的方法（用于工程设计，仿真和VR/AR）。技术目标是建立软件工具，使艺术家能够在现场舞台表演和互动纪录片中加入传感器驱动的视觉效果。虽然有很多关于使用深度学习解决3D图形问题的研究，但提出的工作在问题和解决方案的制定方式上是不同的和创新的。深度学习解决方案在很大程度上依赖于数据和大量的实验来决定网络架构和超参数。由于大型带注释的数据集很难获得，我的研究方法强调(1)减少对非常大的标记数据的依赖，但在需要时创建足够的数据，(2)在适当访问基础设施的情况下让研究生尽早进行实验，(3)在制定问题时牢记理论基础以及行业和艺术家的最终兴趣，以及(4)在开发创新解决方案的同时跟踪最新技术。具体来说，我将研究将深度学习和机器推理与手工制作的特征相结合，从而减少对大数据和大型神经网络的需求。我将与我的3D图形组的学生一起进行研究，该组具有所需的基础设施和行业合作。平均每年将有3名博士和2名硕士学生接受视觉数据计算前沿研究和技术的培训。以基础研究为主要目标，他们将根据需要在小团队中工作，并获得实现，测试和验证复杂系统的技能，并交流他们的研究结果。特别是博士研究生，将被培养在3D图形方面发展他们自己的长期研究计划。基础研究将通过相关场所的出版物为3D图形添加新的知识。新工具将使加拿大的工业、艺术家、娱乐和虚拟仿真部门受益。

My research program is primarily in three dimensional (3D) graphics. The overarching objective is to efficiently synthesize digital objects and worlds (virtual or augmented reality - VR/AR) and interact with them easily. The current focus is on using sensed data and graphics processing (GPU) for the development of data-driven methods, with application in games, entertainment, engineering and virtual simulation. While earlier work was mainly on algorithmic methods using hand-crafted features, the main idea in the proposal here is to develop new methods which apply machine/deep learning and machine inference to solve 3D graphics problems in 3D reconstruction and human motion. The proposed research has two core research objectives and one technical objective. The first is creating natural looking human action sequences from motion data (for use in games, films and VR/AR). The second is methods for automatic reconstruction of 3D objects and larger scenes from sensed visual data (for use in engineering design, simulation and VR/AR). The technical objective is to build software tools which can empower artists to include sensor driven visual effects in live stage performances, and in interactive documentaries. While there is a lot of research on using deep learning for 3D graphics problems, the proposed work is different and innovative in the way the problems and solutions are formulated. Deep learning solutions depend heavily on data and on extensive experiments to decide on the network architecture and hyper-parameters. Since large annotated data sets are difficult to obtain, my research methodology emphasizes (1) reducing dependence on very large labelled data, yet creating adequate data if needed, (2) initiating research students early into experimentation with appropriate access to infrastructure, (3) formulating problems keeping in mind theoretical foundations as well as eventual interest to industry and artists, and (4) keeping track of state of the art while developing innovative solutions. Specifically, I will investigate combining deep learning and machine inference with hand-crafted features, thereby reducing the need for large data and large neural networks. I will build upon ongoing research with students in my 3D graphics group which has the required infrastructure and industry collaborations. On average 3 PhD and 2 MSc students will be trained per year in leading edge research and technologies of visual data computing. With basic research pursuits being the primary objective, they will work in small teams, as needed, and acquire the skills to implement, test and validate complex systems, and communicate their research results.  PhD students, in particular will be trained to develop their own long-term research program in 3D graphics. Basic research will add new knowledge to 3D graphics through publications in relevant venues. New tools will benefit industry, artists, and entertainment and virtual simulation sectors in Canada.