Differentiable Physics Simulators for Realistic 3D Reconstruction

点击翻译按钮获取中文摘要

The reconstruction of 3D structure from camera data is among the classical problems in computer vision. Even though methods have made enormous progress over the last years and now entire actions can be reconstructed in 3D from synchronous videos, most methods are limited to reconstructing the surface of the object. In contrast, the goal of the present project is to reconstruct the entire underlying physics directly from the camera data - this means that we want to determine not only the shape of the object but also its physical properties such as mass, elasticity, momentum and internal and external forces. We believe that this will lead to a more complete understanding of the filmed action. This should make it possible to obtain much more complete reconstructions from significantly fewer observations. In addition, the estimated physical simulation should also allow better extrapolations of the action into the future and thus more reliable predictions of how the action will develop. Methodologically, we want to solve the problem both as an inverse problem and with the help of differentiable physics simulators. The core idea of the latter approach is that we use an encoder-decoder network in which the decoder represents the differentiable simulation of the video from simulation parameters and the encoder, which assigns suitable simulation parameters to an input video, is to be learned from training data.