Data Driven Human Motion Synthesis and 3D Reconstruction

我的研究项目主要是三维（3D）图形。总体目标是有效地合成数字对象和世界（虚拟或增强现实- VR/AR），并轻松与它们进行交互。目前的重点是使用感测数据和图形处理（GPU）来开发数据驱动方法，并应用于游戏，娱乐，工程和虚拟仿真。虽然早期的工作主要是使用手工特征的算法方法，但这里的主要思想是开发新方法，应用机器/深度学习和机器推理来解决3D重建和人体运动中的3D图形问题。该研究有两个核心研究目标和一个技术目标。第一个是从运动数据中创建自然的人类动作序列（用于游戏，电影和VR/AR）。第二种是从感知的视觉数据自动重建3D对象和更大场景的方法（用于工程设计，仿真和VR/AR）。技术目标是建立软件工具，使艺术家能够在现场舞台表演和互动纪录片中包括传感器驱动的视觉效果。 虽然有很多关于将深度学习用于3D图形问题的研究，但所提出的工作在问题和解决方案的制定方式上是不同的和创新的。深度学习解决方案在很大程度上依赖于数据和大量的实验来决定网络架构和超参数。由于大型注释数据集很难获得，我的研究方法强调（1）减少对非常大的标记数据的依赖，但如果需要，则创建足够的数据，（2）在适当访问基础设施的情况下，尽早启动研究学生进行实验，（3）在考虑理论基础以及行业和艺术家的最终兴趣的情况下制定问题，以及（4）在开发创新解决方案的同时跟踪最新技术水平。具体来说，我将研究将深度学习和机器推理与手工制作的功能相结合，从而减少对大数据和大型神经网络的需求。我将建立在正在进行的研究与学生在我的3D图形组具有所需的基础设施和行业合作。平均每年将有3名博士和2名硕士学生接受视觉数据计算前沿研究和技术的培训。随着基础研究的追求是主要目标，他们将在小团队工作，根据需要，并获得实施，测试和验证复杂系统的技能，并传达他们的研究成果。 特别是博士生将接受培训，以开发自己的3D图形长期研究计划。基础研究将通过相关场所的出版物为3D图形增添新知识。新工具将使加拿大的工业、艺术家、娱乐和虚拟仿真部门受益。

My research program is primarily in three dimensional (3D) graphics. The overarching objective is to efficiently synthesize digital objects and worlds (virtual or augmented reality - VR/AR) and interact with them easily. The current focus is on using sensed data and graphics processing (GPU) for the development of data-driven methods, with application in games, entertainment, engineering and virtual simulation. While earlier work was mainly on algorithmic methods using hand-crafted features, the main idea in the proposal here is to develop new methods which apply machine/deep learning and machine inference to solve 3D graphics problems in 3D reconstruction and human motion. The proposed research has two core research objectives and one technical objective. The first is creating natural looking human action sequences from motion data (for use in games, films and VR/AR). The second is methods for automatic reconstruction of 3D objects and larger scenes from sensed visual data (for use in engineering design, simulation and VR/AR). The technical objective is to build software tools which can empower artists to include sensor driven visual effects in live stage performances, and in interactive documentaries. While there is a lot of research on using deep learning for 3D graphics problems, the proposed work is different and innovative in the way the problems and solutions are formulated. Deep learning solutions depend heavily on data and on extensive experiments to decide on the network architecture and hyper-parameters. Since large annotated data sets are difficult to obtain, my research methodology emphasizes (1) reducing dependence on very large labelled data, yet creating adequate data if needed, (2) initiating research students early into experimentation with appropriate access to infrastructure, (3) formulating problems keeping in mind theoretical foundations as well as eventual interest to industry and artists, and (4) keeping track of state of the art while developing innovative solutions. Specifically, I will investigate combining deep learning and machine inference with hand-crafted features, thereby reducing the need for large data and large neural networks. I will build upon ongoing research with students in my 3D graphics group which has the required infrastructure and industry collaborations. On average 3 PhD and 2 MSc students will be trained per year in leading edge research and technologies of visual data computing. With basic research pursuits being the primary objective, they will work in small teams, as needed, and acquire the skills to implement, test and validate complex systems, and communicate their research results.  PhD students, in particular will be trained to develop their own long-term research program in 3D graphics. Basic research will add new knowledge to 3D graphics through publications in relevant venues. New tools will benefit industry, artists, and entertainment and virtual simulation sectors in Canada.