CHS: Medium: Collaborative Research: Physics and Learning Integration Using differentiable rendering

CHS：媒介：协作研究：使用可微渲染的物理和学习集成

• 批准号: 1900783
• 负责人: Kavita Bala
• 金额: $ 40万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900783&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Physics

Using image measurements to understand and acquire properties of the physical world (such as the shape of an object, the reflectance of a surface, or the lighting in a room) is a critical capability for many sciences including medicine, material fabrication, remote sensing, robotics, autonomous navigation, architectural design, computer graphics, and computer vision. At a high level, these properties can be found by taking images and using computational algorithms to infer unknown parameters from the measurements. Broadly, the inference algorithms can be classified into two categories. On the one hand, physics-based algorithms try to analytically model and then invert the physics underlying the process of how a scene of certain parameters produces measured images; these algorithms are generally accurate but require a lot of computation. On the other hand, data-driven algorithms use supervised datasets to learn how to directly map measurements to unknowns; these algorithms are computationally efficient but are not guaranteed to produce accurate predictions. This project aims to transform physical acquisition pipelines, by creating general-purpose computational tools that combine the advantages of physics-based and machine-learning-based techniques, and that are simultaneously efficient, accurate and robust. By developing the theory and computational tools for this integration of simulation and learning, the project has the potential for transformative impact in application areas like industrial quality control, material science, oceanography, and biomedical imaging. Widespread adoption of project outcomes will be encouraged by making new software publicly available, as well as by offering tutorials and workshops in computer graphics, vision, and imaging conferences. The project also includes an education and outreach program that is tightly coupled to the research objectives, and which takes the form of courses, summer workshops, and lab visits for K-12 students intended to introduce them to science at an early stage and encourage STEM education. Additionally, the project will contribute towards broadening participation in computing through targeted involvement in existing programs in the participating institutions that focus on outreach to female students, first-generation students, and students from traditionally underrepresented minorities.This project aims to transform physical acquisition pipelines by creating general-purpose computational tools that enable efficient and robust inference. This will be achieved by coupling physics-based and learning-based approaches, in a way that combines their complementary strengths of accuracy, generality, and efficiency. Three core areas of research will contribute to this. First, the project will develop inference pipelines that synergistically combine neural networks with analysis by synthesis optimization, in order to efficiently produce high-fidelity estimates of physical parameters. Neural networks will be trained in a physics-aware manner, by using physically-accurate renderers as layers in their architecture; this will allow the neural networks to simultaneously leverage supervised information and physical knowledge when making predictions. Second, a new class of physically accurate differentiable renderers will be created, which will enable this tight integration of physics and learning without the need to sacrifice computational efficiency. Instead of images, differentiable renderers will estimate their derivatives with respect to scene parameters; this estimation will be performed in a physically accurate way, using physical simulation algorithms derived from first principles and benefiting from innovations targeting improved efficiency. Finally, the advantages of the developed inference tools will be demonstrated in a diverse range of applications such as autonomous sensing, material science and fabrication, and biomedical imaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用图像测量来了解和获取物理世界的属性(如物体的形状、表面的反射率或房间内的照明)是许多科学的关键能力，包括医学、材料制造、遥感、机器人、自主导航、建筑设计、计算机图形学和计算机视觉。在更高的层面上，这些特性可以通过拍摄图像并使用计算算法从测量中推断未知参数来找到。广义地讲，推理算法可以分为两类。一方面，基于物理的算法试图对特定参数的场景如何产生测量图像的过程进行解析建模，然后反转；这些算法通常是准确的，但需要大量的计算。另一方面，数据驱动算法使用监督数据集来学习如何将测量直接映射到未知数；这些算法在计算上效率很高，但不能保证产生准确的预测。该项目旨在通过创建结合了基于物理的技术和基于机器学习的技术的优点，同时又高效、准确和健壮的通用计算工具，来改造物理获取管道。通过开发这种模拟和学习相结合的理论和计算工具，该项目有可能在工业质量控制、材料科学、海洋学和生物医学成像等应用领域产生革命性的影响。通过向公众提供新软件以及提供计算机图形学、视觉和成像会议的教程和研讨会，将鼓励广泛采用项目成果。该项目还包括一个与研究目标紧密结合的教育和推广计划，其形式包括课程、暑期研讨会和K-12学生的实验室访问，旨在让他们在早期阶段接触科学，并鼓励STEM教育。此外，该项目将通过有针对性地参与参与机构的现有方案，帮助扩大对计算的参与，这些方案侧重于接触女学生、第一代学生和来自传统上代表性较低的少数民族的学生。该项目旨在通过创建能够实现高效和稳健推理的通用计算工具来改变物理获取管道。这将通过将基于物理的方法和基于学习的方法相结合来实现，这种方法结合了它们在准确性、普遍性和效率方面的互补优势。三个核心研究领域将对此作出贡献。首先，该项目将开发推理管道，将神经网络与合成优化分析协同结合，以便高效地产生高保真的物理参数估计。神经网络将以物理感知的方式进行训练，使用物理上精确的呈现器作为其体系结构中的层；这将允许神经网络在进行预测时同时利用受监督的信息和物理知识。其次，将创建一种新的物理上精确的可微渲染器，这将使物理和学习的紧密结合成为可能，而不需要牺牲计算效率。与图像不同，可区分的渲染器将估计它们相对于场景参数的导数；这种估计将以物理准确的方式执行，使用源自基本原理的物理模拟算法，并受益于旨在提高效率的创新。最后，开发的推理工具的优势将在各种应用中得到展示，如自主传感、材料科学和制造以及生物医学成像。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Change-Aware Sampling and Contrastive Learning for Satellite Images

• DOI: 10.1109/cvpr52729.2023.00509
• 发表时间: 2023-06
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Utkarsh Mall;Bharath Hariharan;Kavita Bala

Path-space differentiable rendering

• DOI: 10.1145/3386569.3392383
• 发表时间: 2020-07
• 期刊: ACM Transactions on Graphics (TOG)
• 作者: Cheng Zhang;Bailey Miller;Kai Yan;Ioannis Gkioulekas;Shuang Zhao

Field-Guide-Inspired Zero-Shot Learning

• DOI: 10.1109/iccv48922.2021.00941
• 发表时间: 2021-08
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Utkarsh Mall;Bharath Hariharan;Kavita Bala

Discovering Underground Maps from Fashion

• DOI: 10.1109/wacv51458.2022.00057
• 发表时间: 2020-12
• 期刊: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Utkarsh Mall;Kavita Bala;Tamara L. Berg;K. Grauman

A differential theory of radiative transfer

• DOI: 10.1145/3355089.3356522
• 发表时间: 2019-11
• 期刊: ACM Transactions on Graphics (TOG)
• 作者: Cheng Zhang;Lifan Wu;Changxi Zheng;Ioannis Gkioulekas;R. Ramamoorthi;Shuang Zhao