CHS: Medium: Collaborative Research: Physics and Learning Integration Using Differentiable Rendering

使用图像测量来理解和获取物理世界的属性（例如物体的形状，表面的反射率或房间中的照明）是许多科学的关键能力，包括医学，材料制造，遥感，机器人，自主导航，建筑设计，计算机图形学和计算机视觉。在高水平上，这些属性可以通过拍摄图像并使用计算算法从测量结果中推断未知参数来找到。广义上，推理算法可以分为两类。一方面，基于物理的算法试图分析建模，然后反转某些参数的场景如何产生测量图像的过程的物理基础;这些算法通常是准确的，但需要大量的计算。另一方面，数据驱动的算法使用监督数据集来学习如何直接将测量值映射到未知值;这些算法在计算上是高效的，但不能保证产生准确的预测。 该项目旨在通过创建通用计算工具来改造物理采集管道，这些工具联合收割机了基于物理和基于机器学习的技术的优势，同时具有高效，准确和鲁棒性。通过开发模拟和学习集成的理论和计算工具，该项目有可能在工业质量控制，材料科学，海洋学和生物医学成像等应用领域产生变革性影响。项目成果的广泛采用将通过公开提供新软件以及提供计算机图形学，视觉和成像会议的教程和研讨会来鼓励。 该项目还包括一个与研究目标紧密结合的教育和推广计划，该计划采取课程，夏季研讨会和K-12学生实验室访问的形式，旨在在早期阶段向他们介绍科学并鼓励STEM教育。此外，该项目将通过有针对性地参与机构的现有计划，扩大对计算的参与，这些计划侧重于向女学生，第一代学生和传统上代表性不足的少数民族学生提供服务。该项目旨在通过创建通用计算工具来改变物理获取管道，从而实现高效和强大的推理。这将通过结合基于物理和基于学习的方法来实现，结合它们在准确性，通用性和效率方面的互补优势。三个核心研究领域将对此做出贡献。 首先，该项目将开发推理管道，协同联合收割机神经网络与综合优化分析相结合，以有效地产生物理参数的高保真估计。神经网络将以物理感知的方式进行训练，通过使用物理精确的渲染器作为其架构中的层;这将允许神经网络在进行预测时同时利用监督信息和物理知识。 其次，将创建一类新的物理精确可微渲染器，这将使物理和学习紧密结合，而无需牺牲计算效率。可微分渲染器将估计它们相对于场景参数的导数，而不是图像;这种估计将以物理精确的方式进行，使用从第一原理导出的物理模拟算法，并受益于针对提高效率的创新。 最后，开发的推理工具的优势将在自主传感、材料科学和制造以及生物医学成像等各种应用中得到证明。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Using image measurements to understand and acquire properties of the physical world (such as the shape of an object, the reflectance of a surface, or the lighting in a room) is a critical capability for many sciences including medicine, material fabrication, remote sensing, robotics, autonomous navigation, architectural design, computer graphics, and computer vision. At a high level, these properties can be found by taking images and using computational algorithms to infer unknown parameters from the measurements. Broadly, the inference algorithms can be classified into two categories. On the one hand, physics-based algorithms try to analytically model and then invert the physics underlying the process of how a scene of certain parameters produces measured images; these algorithms are generally accurate but require a lot of computation. On the other hand, data-driven algorithms use supervised datasets to learn how to directly map measurements to unknowns; these algorithms are computationally efficient but are not guaranteed to produce accurate predictions. This project aims to transform physical acquisition pipelines, by creating general-purpose computational tools that combine the advantages of physics-based and machine-learning-based techniques, and that are simultaneously efficient, accurate and robust. By developing the theory and computational tools for this integration of simulation and learning, the project has the potential for transformative impact in application areas like industrial quality control, material science, oceanography, and biomedical imaging. Widespread adoption of project outcomes will be encouraged by making new software publicly available, as well as by offering tutorials and workshops in computer graphics, vision, and imaging conferences. The project also includes an education and outreach program that is tightly coupled to the research objectives, and which takes the form of courses, summer workshops, and lab visits for K-12 students intended to introduce them to science at an early stage and encourage STEM education. Additionally, the project will contribute towards broadening participation in computing through targeted involvement in existing programs in the participating institutions that focus on outreach to female students, first-generation students, and students from traditionally underrepresented minorities.This project aims to transform physical acquisition pipelines by creating general-purpose computational tools that enable efficient and robust inference. This will be achieved by coupling physics-based and learning-based approaches, in a way that combines their complementary strengths of accuracy, generality, and efficiency. Three core areas of research will contribute to this. First, the project will develop inference pipelines that synergistically combine neural networks with analysis by synthesis optimization, in order to efficiently produce high-fidelity estimates of physical parameters. Neural networks will be trained in a physics-aware manner, by using physically-accurate renderers as layers in their architecture; this will allow the neural networks to simultaneously leverage supervised information and physical knowledge when making predictions. Second, a new class of physically accurate differentiable renderers will be created, which will enable this tight integration of physics and learning without the need to sacrifice computational efficiency. Instead of images, differentiable renderers will estimate their derivatives with respect to scene parameters; this estimation will be performed in a physically accurate way, using physical simulation algorithms derived from first principles and benefiting from innovations targeting improved efficiency. Finally, the advantages of the developed inference tools will be demonstrated in a diverse range of applications such as autonomous sensing, material science and fabrication, and biomedical imaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.