Differentiable Physics Simulators for Realistic 3D Reconstruction

从摄像机数据重建三维结构是计算机视觉中的经典问题之一。尽管在过去的几年里，方法取得了巨大的进步，现在可以从同步视频中以3D方式重建整个动作，但大多数方法仅限于重建对象的表面。相比之下，本项目的目标是直接从相机数据重建整个基础物理--这意味着我们不仅想要确定物体的形状，还想确定它的物理属性，如质量、弹性、动量和内力和外力。我们相信，这将使人们对所拍摄的动作有更完整的了解。这应该可以从更少的观测中获得更完整的重建。此外，估计的物理模拟还应允许更好地推断未来的行动，从而更可靠地预测行动将如何发展。在方法论上，我们希望将问题作为反问题来解决，并借助可微物理模拟器来解决。后一种方法的核心思想是使用编解码器网络，其中解码器表示来自仿真参数的视频的可区分仿真，而编码器从训练数据中学习为输入视频分配合适的仿真参数。

The reconstruction of 3D structure from camera data is among the classical problems in computer vision. Even though methods have made enormous progress over the last years and now entire actions can be reconstructed in 3D from synchronous videos, most methods are limited to reconstructing the surface of the object. In contrast, the goal of the present project is to reconstruct the entire underlying physics directly from the camera data - this means that we want to determine not only the shape of the object but also its physical properties such as mass, elasticity, momentum and internal and external forces. We believe that this will lead to a more complete understanding of the filmed action. This should make it possible to obtain much more complete reconstructions from significantly fewer observations. In addition, the estimated physical simulation should also allow better extrapolations of the action into the future and thus more reliable predictions of how the action will develop. Methodologically, we want to solve the problem both as an inverse problem and with the help of differentiable physics simulators. The core idea of the latter approach is that we use an encoder-decoder network in which the decoder represents the differentiable simulation of the video from simulation parameters and the encoder, which assigns suitable simulation parameters to an input video, is to be learned from training data.