Data Driven Human Motion Synthesis and 3D Reconstruction

我的研究项目主要是三维(3D)图形。首要目标是有效地合成数字对象和世界(虚拟或增强现实-VR/AR)，并轻松地与它们交互。目前的重点是使用传感数据和图形处理(GPU)来开发数据驱动的方法，并在游戏、娱乐、工程和虚拟仿真中应用。虽然早期的工作主要是使用手工制作的特征的算法方法，但这里的主要思想是开发新的方法，应用机器/深度学习和机器推理来解决3D重建和人体运动中的3D图形问题。拟议的研究有两个核心研究目标和一个技术目标。第一种是从运动数据创建看起来自然的人类动作序列(用于游戏、电影和VR/AR)。第二种是从感知的视觉数据自动重建3D对象和更大场景的方法(用于工程设计、模拟和VR/AR)。技术目标是构建软件工具，使艺术家能够在现场舞台表演和互动纪录片中加入传感器驱动的视觉效果。 虽然已经有很多关于深度学习用于3D图形问题的研究，但所提出的工作在问题和解决方案的制定方式上是不同的和创新的。深度学习解决方案在很大程度上依赖于数据和广泛的实验来决定网络架构和超级参数。由于很难获得大量的带注释的数据集，我的研究方法强调(1)减少对非常大的标记数据的依赖，但在需要时创建足够的数据，(2)让研究学生及早开始尝试适当地访问基础设施，(3)在考虑到理论基础以及行业和艺术家最终的兴趣的情况下制定问题，以及(4)在开发创新解决方案的同时跟踪最新技术。具体地说，我将研究将深度学习和机器推理与手工制作的特征相结合，从而减少对大型数据和大型神经网络的需求。我将与我的3D图形小组的学生一起进行正在进行的研究，该小组拥有必要的基础设施和行业合作。平均每年将有3名博士和2名硕士研究生接受视觉数据计算前沿研究和技术方面的培训。以基础研究为主要目标，他们将根据需要以小组形式工作，并获得实施、测试和验证复杂系统的技能，并交流他们的研究结果。特别是博士生，他们将接受培训，开发自己的3D图形长期研究计划。基础研究将通过相关场馆的出版物为3D图形增加新的知识。新工具将使加拿大的工业、艺术家、娱乐和虚拟模拟部门受益。

My research program is primarily in three dimensional (3D) graphics. The overarching objective is to efficiently synthesize digital objects and worlds (virtual or augmented reality - VR/AR) and interact with them easily. The current focus is on using sensed data and graphics processing (GPU) for the development of data-driven methods, with application in games, entertainment, engineering and virtual simulation. While earlier work was mainly on algorithmic methods using hand-crafted features, the main idea in the proposal here is to develop new methods which apply machine/deep learning and machine inference to solve 3D graphics problems in 3D reconstruction and human motion. The proposed research has two core research objectives and one technical objective. The first is creating natural looking human action sequences from motion data (for use in games, films and VR/AR). The second is methods for automatic reconstruction of 3D objects and larger scenes from sensed visual data (for use in engineering design, simulation and VR/AR). The technical objective is to build software tools which can empower artists to include sensor driven visual effects in live stage performances, and in interactive documentaries. While there is a lot of research on using deep learning for 3D graphics problems, the proposed work is different and innovative in the way the problems and solutions are formulated. Deep learning solutions depend heavily on data and on extensive experiments to decide on the network architecture and hyper-parameters. Since large annotated data sets are difficult to obtain, my research methodology emphasizes (1) reducing dependence on very large labelled data, yet creating adequate data if needed, (2) initiating research students early into experimentation with appropriate access to infrastructure, (3) formulating problems keeping in mind theoretical foundations as well as eventual interest to industry and artists, and (4) keeping track of state of the art while developing innovative solutions. Specifically, I will investigate combining deep learning and machine inference with hand-crafted features, thereby reducing the need for large data and large neural networks. I will build upon ongoing research with students in my 3D graphics group which has the required infrastructure and industry collaborations. On average 3 PhD and 2 MSc students will be trained per year in leading edge research and technologies of visual data computing. With basic research pursuits being the primary objective, they will work in small teams, as needed, and acquire the skills to implement, test and validate complex systems, and communicate their research results. PhD students, in particular will be trained to develop their own long-term research program in 3D graphics. Basic research will add new knowledge to 3D graphics through publications in relevant venues. New tools will benefit industry, artists, and entertainment and virtual simulation sectors in Canada.