RI: Small: Semantic 3D Neural Rendering Field Models that are Accurate, Complete, Flexible, and Scalable

该项目将研究从多个图像创建场景模型的方法，该模型可以实现可视化，合成，计数，测量和其他分析。该项目的目标是由统一的几何（在哪里，什么形状，多大）和语义（它是什么，它是什么样的）场景模型的需求驱动的，基于调查人员在建筑管理和车辆安全产品的直接经验。到目前为止，计算机视觉可以说在互联网领域产生了最大的影响。这个项目需要更广泛的应用，涉及物理世界，其潜在的影响是很难夸大的。由此产生的能力将为实时建模、增强现实、仿真和机器人应用奠定基础。该项目为可查询、可编辑和可操作的语义和几何场景模型奠定了基础，这是计算机视觉中的一个基础问题。神经渲染领域，视觉语言模型和扩散已经令人印象深刻地证明了单独的图像合成和分析应用。该项目将这些进步结合在一起，为3D语义场景建模提供新的表示和功能。其结果是一种可扩展的和强大的方法来创建，更新，查询和编辑从多个观察推断的世界模型。具体而言，该项目涉及三项行动计划。 首先是创建可测量和可网格化的3D场景模型，这些模型可以从稀疏视图中有效地估计，并扩展到数千张图像。这包括几个方面的发展：新的高效优化，紧凑的表示法;纳入单目几何估计;姿态，增益和其他参数的联合细化;无缝扩展到大规模场景和照片集的方法;以及提取高分辨率网格，地板地图和其他常见交付物的方法。第二个行动计划是纳入语义信息和解码器，用于计数、测量和变化检测。这包括在连续嵌入中编码语义，并创建解码器，用于可视化，计数，测量和其他场景范围的几何语义查询，以实现实时，灵活的映射和设施评估。 第三个行动计划是通过整合生成和预测过程，在直接观测之外进行推断，并在新观测到达时推断和更新模型。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

This project will investigate methods to create, from multiple images, a scene model that enables visualization, synthesis, counting, measurement, and other analysis. The goals of the project are driven by the need for unified geometric (where, what shape, how big) and semantic (what is it, what is it like) scene models, based on the investigators' direct experience in building products for construction management and vehicle safety. So far, computer vision has arguably had its largest impact in internet domains. This project is needed for broader applications involving the physical world, and the potential impact is hard to overstate. Resulting capabilities will lay foundations for real-time modeling, augmented reality, simulation, and robotics applications. The project lays the groundwork for a queryable, editable, and actionable semantic and geometric scene model, a foundational problem in computer vision. Neural rendering fields, vision language models, and diffusion have been impressively demonstrated for separate image synthesis and analysis applications. The project brings these advances together to enable new representations and capabilities for 3D semantic scene modeling. The result is a scalable and robust approach to create, update, query, and edit models of the world inferred from multiple observations. In particular, the project involves three plans of action. The first is to create measurable and meshable 3D scene models that can be efficiently estimated from sparse views and scale to thousands of images. This includes several developments: new efficiently optimizable, compact representations; incorporation of monocular geometry estimates; joint refinement of pose, gain, and other parameters; methods to scale seamlessly to massive scenes and photo sets; and ways to extract high resolution meshes, floor maps, and other common deliverables. The second plan of action is to incorporate semantic information and decoders for counting, measuring, and change detection. This includes encoding semantics in continuous embeddings and creating decoders for visualizing, counting, measuring, and other scene-wide geometric-semantic queries, to enable real-time, flexible mapping and facility assessment. The third plan of action is to extrapolate beyond direct observations and infer and update models as new observations arrive by integrating generative and predictive processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.