权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

RI: Small: Physically-Based Learning for Shape, Lighting and Material in Complex Indoor Scenes

RI：小型：复杂室内场景中形状、照明和材质的基于物理的学习

基本信息

批准号：
2110409
负责人：
Manmohan Chandraker
金额：
$ 50万
依托单位：
University of California-San Diego
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2021
资助国家：
美国
起止时间：
2021-10-01 至 2024-09-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2110409&HistoricalAwards=false
关键词：
RI Small Physically Based Learning

项目摘要

Indoor scenes involve visible and invisible light sources interacting with spatially-varying materials and shapes, together with interreflections and shadows, to produce images that display complex local variations. Applications such as augmented reality require editing an indoor scene to insert new objects, changing materials in some parts of the scene, or visualizing a room under a different illumination. Estimating those factors of image formation constitutes the problem of inverse rendering, which is especially ill-posed when the input is only a few images of an indoor scene acquired with a commodity camera. While traditional measurement-based methods need expensive devices and do not scale to large-scale scenes, conventional learning methods suffer from a lack of data and expressive power to handle the above complex visual effects. This research addresses the longstanding computer vision and graphics challenge of inverse rendering through a holistic approach of modeling complex materials and illumination, developing novel physically inspired deep networks and generating large-scale training data. The project will have a transformative effect on industry and society through photorealistic augmented reality and image editing applications such as view synthesis, object insertion, material, and light source editing, with accurate shadows and interreflections. The project also incorporates experiential insights from the above technological advances into college and K-12 education through a new interdisciplinary curriculum in computer vision and computer graphics.This project develops physically based deep networks that incorporate the inductive bias of image formation to achieve generalizable representations that estimate shape, material and especially lighting with unprecedented detail, by accounting for visibilities in a single image, consistency in multiple images and complex light transport in challenging or dynamic scenes. It designs novel modules such as neural rendering layers and reflectance volumes that incorporate the domain knowledge of image formation. It achieves scalability through parsimonious representations that retain descriptive power while generalizing to complex lighting effects such as refraction, scattering and dynamic light transport. It develops large-scale training datasets with realistic spatially varying lighting and complex high-quality material.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

室内场景包括可见和不可见光源与空间变化的材料和形状相互作用，以及相互反射和阴影，以产生显示复杂的局部变化的图像。增强现实等应用程序需要编辑室内场景以插入新对象，改变场景某些部分的材料，或者在不同的照明下可视化房间。对这些图像形成因素的估计构成了逆向渲染的问题，当输入仅是用商用相机获取的室内场景的几幅图像时，这种问题尤其不适定。传统的基于测量的方法需要昂贵的设备，并且不能扩展到大规模的场景，传统的学习方法缺乏数据和表达能力来处理上述复杂的视觉效果。本研究通过对复杂材料和照明建模的整体方法，开发新颖的物理启发深度网络和生成大规模训练数据，解决了长期存在的计算机视觉和图形逆向渲染的挑战。该项目将通过逼真的增强现实和图像编辑应用，如视图合成、物体插入、材料和光源编辑，以及精确的阴影和相互反射，对工业和社会产生变革性影响。该项目还通过计算机视觉和计算机图形学的新跨学科课程，将上述技术进步的经验见解纳入大学和K-12教育。该项目开发了基于物理的深度网络，该网络结合了图像形成的归纳偏差，通过考虑单幅图像的可见性、多幅图像的一致性以及具有挑战性或动态场景中的复杂光传输，实现了具有前所未有细节的形状、材料，特别是照明的一般化表示。它设计了新颖的模块，如神经渲染层和反射率体积，结合了图像形成的领域知识。它通过简洁的表示实现可扩展性，保留了描述能力，同时推广到复杂的照明效果，如折射、散射和动态光传输。它开发了大规模的训练数据集，具有逼真的空间变化照明和复杂的高质量材料。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

期刊论文数量（5）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

NeuSample: Importance Sampling for Neural Materials

DOI：
10.1145/3588432.3591524
发表时间：
2023-07
期刊：
ACM SIGGRAPH 2023 Conference Proceedings
影响因子：
0
作者：
Bing Xu;Liwen Wu;Miloš Hašan;Fujun Luan;Iliyan Georgiev;Zexiang Xu;R. Ramamoorthi
通讯作者：
Bing Xu;Liwen Wu;Miloš Hašan;Fujun Luan;Iliyan Georgiev;Zexiang Xu;R. Ramamoorthi

Physically-Based Editing of Indoor Scene Lighting from a Single Image

DOI：
10.48550/arxiv.2205.09343
发表时间：
2022-05
期刊：
ArXiv
影响因子：
0
作者：
Zhengqin Li;Jia Shi;Sai Bi;Rui Zhu;Kalyan Sunkavalli;Milovs Havsan;Zexiang Xu;R. Ramamoorthi;Manmohan Chandraker
通讯作者：
Zhengqin Li;Jia Shi;Sai Bi;Rui Zhu;Kalyan Sunkavalli;Milovs Havsan;Zexiang Xu;R. Ramamoorthi;Manmohan Chandraker

IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes