Data Driven Human Motion Synthesis and 3D Reconstruction

点击翻译按钮获取中文摘要

My research program is primarily in three dimensional (3D) graphics. The overarching objective is to efficiently synthesize digital objects and worlds (virtual or augmented reality - VR/AR) and interact with them easily. The current focus is on using sensed data and graphics processing (GPU) for the development of data-driven methods, with application in games, entertainment, engineering and virtual simulation. While earlier work was mainly on algorithmic methods using hand-crafted features, the main idea in the proposal here is to develop new methods which apply machine/deep learning and machine inference to solve 3D graphics problems in 3D reconstruction and human motion. The proposed research has two core research objectives and one technical objective. The first is creating natural looking human action sequences from motion data (for use in games, films and VR/AR). The second is methods for automatic reconstruction of 3D objects and larger scenes from sensed visual data (for use in engineering design, simulation and VR/AR). The technical objective is to build software tools which can empower artists to include sensor driven visual effects in live stage performances, and in interactive documentaries. While there is a lot of research on using deep learning for 3D graphics problems, the proposed work is different and innovative in the way the problems and solutions are formulated. Deep learning solutions depend heavily on data and on extensive experiments to decide on the network architecture and hyper-parameters. Since large annotated data sets are difficult to obtain, my research methodology emphasizes (1) reducing dependence on very large labelled data, yet creating adequate data if needed, (2) initiating research students early into experimentation with appropriate access to infrastructure, (3) formulating problems keeping in mind theoretical foundations as well as eventual interest to industry and artists, and (4) keeping track of state of the art while developing innovative solutions. Specifically, I will investigate combining deep learning and machine inference with hand-crafted features, thereby reducing the need for large data and large neural networks. I will build upon ongoing research with students in my 3D graphics group which has the required infrastructure and industry collaborations. On average 3 PhD and 2 MSc students will be trained per year in leading edge research and technologies of visual data computing. With basic research pursuits being the primary objective, they will work in small teams, as needed, and acquire the skills to implement, test and validate complex systems, and communicate their research results.  PhD students, in particular will be trained to develop their own long-term research program in 3D graphics. Basic research will add new knowledge to 3D graphics through publications in relevant venues. New tools will benefit industry, artists, and entertainment and virtual simulation sectors in Canada.