EAGER: Deep Architectures for Ppredicting 3D Object Motion

我们的日常生活环境中充斥着许多功能物体，我们可以通过它们的移动部件与之互动(例如，转椅、笔记本电脑、自行车和汽车，仅举几例)。为了让自主代理在现实环境中正确地与这些对象交互，这些代理必须配备能够将对象解析为其移动部分的算法。但这还不够。通过商用3D传感器和现代3D建模技术的广泛使用，现在可以使用包含数百万日常对象的数字表示的大型存储库(如ShapeNet)，但这些表示大部分目前是静态的，也就是说，它们表示对象的单个快照。为了在动态、虚拟环境和动画应用中使用这些对象表示，需要自动将它们分割成运动部分并为它们合成合理运动的方法。这个项目将探索实现这一目标的新的深度学习架构的设计、实现和测试，从而使大部分静态3D数据集“活起来”。通过改进3D建模和动画软件，新算法将产生广泛的工业影响，而生成的运动数据将用于训练新的计算机视觉算法，用于视频中的对象运动识别和跟踪。要实现项目目标，将需要开发新的算法，通过自动识别3D对象的运动部分并基于来自真实世界的类似对象的输入参考视频对其进行动画处理，以及通过结合新方法来估计部分2D-3D对应关系、将2D运动线索提升到3D以及推断3D形状的运动装备，从而将3D对象的静态数字表示转换为动态数字表示。该项目将被组织成两个主要推动力，每个推力都将提出自己的研究挑战。第一个推力将研究新的深度学习体系结构，用于执行基于移动的3D对象分割并预测其各部分的潜在运动。该建筑将应用于具有刚性移动部件的人造物体。这部分研究将在项目的第一年进行。第二个推力将扩展之前的工作，使代表四足动物、鸟类和鱼(即数字生命数据集中的动物)等生物的3D模型具有动画效果。这些模型经历非刚性变形，因此必须修改体系结构以估计和控制更复杂的变形基本体。这部分工作将在项目的第二年执行。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Our everyday living environments are populated with lots of functional objects with which we can interact through their moving parts (e.g., swivel chairs, laptops, bikes and cars, to name just a few). For autonomous agents to correctly interact with these objects in real-world settings, the agents must be equipped with algorithms that are able to parse the objects into their moving parts. But that is not enough. Through the widespread use of commodity 3D sensors and modern 3D modeling techniques, large repositories (such as ShapeNet) containing millions of digital representations of everyday objects are now available, but these representations are for the most part currently static, that is to say they represent single snapshots of objects. To make use of these object representations in dynamic, virtual environments and in animation applications, methods that automatically segment them into moving parts and synthesize plausible motions for them are needed. This project will explore the design, implementation, and testing of new deep learning architectures that accomplish this, and thereby bring large portions of static 3D datasets "to life." The new algorithms will have broad industrial impact by advancing 3D modeling and animation software, while the generated motion data will be useful for training new computer vision algorithms for object motion recognition and tracking in videos.Achieving the project goals will require development of new algorithms to convert static digital representations of 3D objects into dynamic ones by automatically recognizing their moving parts and animating them based on input reference videos of similar objects from the real world and through incorporation of novel methods for estimating partial 2D-3D correspondences, for lifting 2D motion cues to 3D, and for inferring motion rigs for 3D shapes. The project will be organized into two main thrusts, each of which will present its own research challenges. The first thrust will investigate new deep learning architectures for performing mobility-based segmentation of 3D objects and predicting the underlying motion of their parts. The architecture will be applied to man-made objects with rigidly moving parts. This part of the research will be carried out in the first year of the project. The second thrust will extend the previous work to animate 3D models representing living organisms such as quadrupeds, birds and fish (i.e., animals from the DigitalLife dataset). These models undergo non-rigid deformations, so the architecture will have to be modified to estimate and control more sophisticated deformation primitives. This part of the work will be executed in the second year of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.