CAREER: Deep Learning Empowered Nonlinear Deformable Model

世界上的一切都会变形，因此建模高质量的变形成为严肃和现实主义驱动的视觉应用的核心算法成分，例如高保真动画，虚拟现实，医疗数据分析，手术模拟和数字制造/原型制作，仅举几例。虽然变形已经研究了几十年，但变形模拟因其昂贵的计算而臭名昭着。随着复杂传感设备和采集技术的快速发展，数据的复杂性、规模和维数呈指数级增长，大规模几何体在现代三维数据处理中变得无处不在。即使使用最先进的硬件，大规模的可变形模拟仍然需要数小时，数天甚至数周。在这个数据爆炸的时代，对计算效率和仿真真实性的要求越来越高，这对这个经典的计算问题提出了前所未有的挑战，因此需要针对大规模，复杂和非线性可变形模型的改变游戏规则的算法技术来支持未来的图形应用。 如果成功，该项目不仅将拓展基于物理的仿真技术的前沿，还将深刻激发图形之外的更广泛的计算社区，并实现各种应用。在变形模拟过程中，为了跟踪变形体的连续形状演化，需要反复求解一个非线性系统。具有复杂几何形状的可变形物体可以容纳大量未知的自由度，并且由此产生的高维集成变得令人望而却步。为了克服这个问题，该项目将开发一个重新命名的可变形模型，该模型系统地集成了先进的仿真技术和深度学习（DL）工具，特别是深度神经网络（DNN）。假设数字仿真为我们提供了几乎无限的无噪声训练数据，应该充分利用和利用这些数据来帮助我们应对看不见的但困难的仿真或计算挑战。与现有的用封闭形式的公式解释数据的数据驱动方法（例如，使用凸插值），DNN提供了一种通用的机制，可以以端到端的方式提取隐藏在原始数据背后的内在特征，并且已经在许多长期存在的计算机视觉问题（如对象检测，分类和注释）中展示了显着的成果。然而，在基于物理的仿真中利用DL并不容易。虽然理论上仍然可以使用非常高维的输入向量对所有这些参数进行编码，但相应的网络将非常庞大和复杂。即使我们设法收集足够的训练数据来优化这个网络，它的一个向前传递可能比传统的模拟器慢，使DL完全无利可图。在这个项目中，我们将彻底调查这些重大的技术挑战，为数据驱动的可变形模拟建立一个数据结构和算法技术的集合，从而为基于DL的物理模拟到下一代计算机图形学铺平道路。这个奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Everything in the world deforms, so modeling high-quality deformations becomes a core algorithmic ingredient for serious and realism-driven visual applications such as high-fidelity animation, virtual reality, medical data analysis, surgical simulation, and digital fabrication/prototyping, to name just a few. While deformation has been studied for decades, deformable simulation is notorious for its costly computation. With the rapid development of sophisticated sensing devices and acquisition techniques, the complexities, scales and dimensionalities of the data have grown exponentially, and large-scale geometries are becoming ubiquitous in modern 3D data processing. Even with state-of-the-art hardware, a massive deformable simulation can still take hours, days, or even weeks. In this era of data explosion, increasing demands on both computing efficiency and simulation realism impose unprecedented challenges on this classic computing problem, so game-changing algorithmic techniques for large-scale, complex, and nonlinear deformable models are needed to empower future graphics applications. If successful, this project will not only expand the frontier of physics-based simulation technologies, but also profoundly inspire broader computing communities beyond graphics and enable a variety of applications. During a deformable simulation, a nonlinear system needs to be repetitively solved in order to track the continuous shape evolution of the deforming body. A deformable object with complex geometry could house a large number of unknown degrees of freedom, and the resulting high-dimensional integration becomes prohibitive. To overcome this problem, this project will develop a re-branded deformable model which systematically integrates advanced simulation techniques and deep learning (DL) tools, specifically deep neural networks (DNNs). The hypothesis is that digital simulation provides us nearly unlimited noise-free training data, which should be fully exploited and leveraged to benefit unseen yet difficult simulation or computing challenges. Unlike existing data-driven methods that interpret the data with a closed-form formulation (e.g., using a convex interpolation), DNNs provide a universal mechanism to extract intrinsic features hidden behind the raw data in an end-to-end manner, and have already demonstrated significant outcomes in many long-standing computer vision problems like object detection, classification, and annotation. However, harnessing DL in physics-based simulation is not easy. While in theory one may still encode all of these parameters using a very high-dimensional input vector, the corresponding network would be extremely large and complex. Even if we manage to collect sufficient training data to optimize this net, a single forward pass of it may be slower than a conventional simulator, making DL completely unprofitable. In this project, we will thoroughly investigate those grand technical challenges, forge a collection of data structures and algorithmic techniques for the data-driven deformable simulation, and thereby pave the way for DL-based physics simulation to next-generation computer graphics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.