CAREER: Deep Learning Empowered Nonlinear Deformable Model

世界上的一切都会变形，因此高质量的变形建模成为严肃和逼真的视觉应用程序的核心算法成分，例如高保真动画、虚拟现实、医疗数据分析、手术模拟和数字制造/原型，仅举几例。虽然对变形的研究已经有几十年了，但可变形模拟因其昂贵的计算而臭名昭著。随着尖端传感设备和采集技术的快速发展，数据的复杂性、规模和维度呈指数级增长，大尺度几何在现代三维数据处理中变得无处不在。即使使用最先进的硬件，大规模可变形模拟仍然需要数小时、数天甚至数周的时间。在这个数据爆炸的时代，对计算效率和模拟真实感的日益增长的要求对这一经典计算问题提出了前所未有的挑战，因此需要针对大规模、复杂和非线性变形模型的改变游戏规则的算法技术来支持未来的图形应用。如果成功，该项目不仅将扩大基于物理的模拟技术的前沿，还将深刻地激励图形以外的更广泛的计算社区，并使各种应用程序成为可能。在可变形模拟中，为了跟踪变形体的连续形状演化，需要对一个非线性系统进行反复求解。具有复杂几何形状的可变形对象可能包含大量未知自由度，由此产生的高维积分变得令人望而却步。为了克服这一问题，该项目将开发一种重新命名的可变形模型，该模型系统地集成了先进的模拟技术和深度学习(DL)工具，特别是深度神经网络(DNN)。我们的假设是，数字仿真为我们提供了几乎无限的无噪声训练数据，这些数据应该得到充分的开发和利用，以造福于看不见但困难的仿真或计算挑战。不同于现有的以封闭形式(例如，使用凸插值)来解释数据的数据驱动方法，DNN提供了一种通用机制来以端到端的方式提取隐藏在原始数据背后的内在特征，并且已经在许多长期存在的计算机视觉问题(如目标检测、分类和注释)中展示了重要的结果。然而，在基于物理的模拟中利用动态链接库并不容易。虽然在理论上，人们仍然可以使用非常高维的输入向量来编码所有这些参数，但相应的网络将非常大和复杂。即使我们设法收集了足够的训练数据来优化这个网络，它的一次向前传递可能比传统的模拟器慢，使DL完全无利可图。在这个项目中，我们将深入调查这些重大的技术挑战，为数据驱动的可变形模拟打造一系列数据结构和算法技术，从而为基于数字图书馆的物理模拟到下一代计算机图形铺平道路。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Everything in the world deforms, so modeling high-quality deformations becomes a core algorithmic ingredient for serious and realism-driven visual applications such as high-fidelity animation, virtual reality, medical data analysis, surgical simulation, and digital fabrication/prototyping, to name just a few. While deformation has been studied for decades, deformable simulation is notorious for its costly computation. With the rapid development of sophisticated sensing devices and acquisition techniques, the complexities, scales and dimensionalities of the data have grown exponentially, and large-scale geometries are becoming ubiquitous in modern 3D data processing. Even with state-of-the-art hardware, a massive deformable simulation can still take hours, days, or even weeks. In this era of data explosion, increasing demands on both computing efficiency and simulation realism impose unprecedented challenges on this classic computing problem, so game-changing algorithmic techniques for large-scale, complex, and nonlinear deformable models are needed to empower future graphics applications. If successful, this project will not only expand the frontier of physics-based simulation technologies, but also profoundly inspire broader computing communities beyond graphics and enable a variety of applications. During a deformable simulation, a nonlinear system needs to be repetitively solved in order to track the continuous shape evolution of the deforming body. A deformable object with complex geometry could house a large number of unknown degrees of freedom, and the resulting high-dimensional integration becomes prohibitive. To overcome this problem, this project will develop a re-branded deformable model which systematically integrates advanced simulation techniques and deep learning (DL) tools, specifically deep neural networks (DNNs). The hypothesis is that digital simulation provides us nearly unlimited noise-free training data, which should be fully exploited and leveraged to benefit unseen yet difficult simulation or computing challenges. Unlike existing data-driven methods that interpret the data with a closed-form formulation (e.g., using a convex interpolation), DNNs provide a universal mechanism to extract intrinsic features hidden behind the raw data in an end-to-end manner, and have already demonstrated significant outcomes in many long-standing computer vision problems like object detection, classification, and annotation. However, harnessing DL in physics-based simulation is not easy. While in theory one may still encode all of these parameters using a very high-dimensional input vector, the corresponding network would be extremely large and complex. Even if we manage to collect sufficient training data to optimize this net, a single forward pass of it may be slower than a conventional simulator, making DL completely unprofitable. In this project, we will thoroughly investigate those grand technical challenges, forge a collection of data structures and algorithmic techniques for the data-driven deformable simulation, and thereby pave the way for DL-based physics simulation to next-generation computer graphics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.