RI: Small: Semantic 3D Neural Rendering Field Models that are Accurate, Complete, Flexible, and Scalable

RI：小型：准确、完整、灵活且可扩展的语义 3D 神经渲染场模型

• 批准号: 2312102
• 负责人: Derek Hoiem
• 金额: $ 60万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-15 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312102&HistoricalAwards=false
• 关键词: RI; Small; Semantic; 3D; Neural

This project will investigate methods to create, from multiple images, a scene model that enables visualization, synthesis, counting, measurement, and other analysis. The goals of the project are driven by the need for unified geometric (where, what shape, how big) and semantic (what is it, what is it like) scene models, based on the investigators' direct experience in building products for construction management and vehicle safety. So far, computer vision has arguably had its largest impact in internet domains. This project is needed for broader applications involving the physical world, and the potential impact is hard to overstate. Resulting capabilities will lay foundations for real-time modeling, augmented reality, simulation, and robotics applications. The project lays the groundwork for a queryable, editable, and actionable semantic and geometric scene model, a foundational problem in computer vision. Neural rendering fields, vision language models, and diffusion have been impressively demonstrated for separate image synthesis and analysis applications. The project brings these advances together to enable new representations and capabilities for 3D semantic scene modeling. The result is a scalable and robust approach to create, update, query, and edit models of the world inferred from multiple observations. In particular, the project involves three plans of action. The first is to create measurable and meshable 3D scene models that can be efficiently estimated from sparse views and scale to thousands of images. This includes several developments: new efficiently optimizable, compact representations; incorporation of monocular geometry estimates; joint refinement of pose, gain, and other parameters; methods to scale seamlessly to massive scenes and photo sets; and ways to extract high resolution meshes, floor maps, and other common deliverables. The second plan of action is to incorporate semantic information and decoders for counting, measuring, and change detection. This includes encoding semantics in continuous embeddings and creating decoders for visualizing, counting, measuring, and other scene-wide geometric-semantic queries, to enable real-time, flexible mapping and facility assessment. The third plan of action is to extrapolate beyond direct observations and infer and update models as new observations arrive by integrating generative and predictive processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将研究从多个图像创建一个可以可视化，综合，计数，测量和其他分析的场景模型创建的方法。该项目的目标是由统一的几何形状（其中，什么形状，多大）和语义（什么是什么，是什么样的）场景模型所驱动的，这是基于调查人员在建造建筑管理和车辆安全的产品方面的直接经验。到目前为止，计算机愿景可以说在互联网领域产生了最大的影响。对于涉及物理世界的更广泛的应用程序，需要该项目，而潜在的影响很难夸大其词。最终的功能将为实时建模，增强现实，仿真和机器人技术应用奠定基础。该项目为可查询，可编辑和可操作的语义和几何场景模型奠定了基础，这是计算机视觉中的基础问题。神经渲染场，视觉语言模型和扩散已被令人印象深刻地证明，以用于单独的图像综合和分析应用。该项目将这些进步融合在一起，以实现3D语义场景建模的新表示和功能。结果是一种可扩展且可靠的方法，可以从多个观察值中推断出世界的世界的创建，更新，查询和编辑模型。特别是，该项目涉及三个行动计划。首先是创建可衡量且可网状的3D场景模型，这些模型可以有效地从稀疏视图和规模缩小到数千个图像。这包括几个发展：新的有效优化，紧凑的表示；掺入单眼几何估计；姿势，增益和其他参数的关节改进；无缝缩放到大型场景和照片集的方法；以及提取高分辨率网格，地板图和其他常见交付的方法。第二种行动计划是结合语义信息和解码器，以计算，测量和变更检测。这包括在连续嵌入中编码语义，并创建用于可视化，计数，测量和其他场景范围内的几何语音查询的解码器，以实现实时，灵活的映射和设施评估。第三个行动计划是通过整合通用和预测过程来推断直接观察以及推断和更新模型，并推断和更新模型。该奖项反映了NSF的法定任务，并通过使用基金会的智力优点和更广泛的影响来诚实地认为，通过评估诚实地支持了审查标准。