RI: Small: Semantic 3D Neural Rendering Field Models that are Accurate, Complete, Flexible, and Scalable

这个项目将研究从多幅图像创建场景模型的方法，该模型能够实现可视化、合成、计数、测量和其他分析。该项目的目标是基于调查人员在建筑施工管理和车辆安全方面的直接经验，对统一的几何(在哪里，什么形状，多大)和语义(它是什么，它是什么样子)场景模型的需求。到目前为止，计算机视觉可以说在互联网领域产生了最大的影响。这个项目对于涉及物理世界的更广泛的应用是必要的，其潜在影响怎么说都不为过。由此产生的能力将为实时建模、增强现实、模拟和机器人应用奠定基础。该项目为可查询、可编辑和可操作的语义和几何场景模型奠定了基础，这是计算机视觉中的一个基本问题。神经渲染领域、视觉语言模型和扩散已经在单独的图像合成和分析应用中得到了令人印象深刻的展示。该项目将这些进步结合在一起，以实现3D语义场景建模的新表示法和功能。其结果是一种可伸缩且健壮的方法来创建、更新、查询和编辑从多个观测推断出的世界模型。特别是，该项目涉及三个行动计划。第一个是创建可测量和可划分网格的3D场景模型，可以从稀疏的视图高效地估计这些模型，并将其缩放到数千幅图像。这包括几项开发：新的高效优化的紧凑表示;结合了单目几何估计;姿势、增益和其他参数的联合细化;无缝缩放到大规模场景和照片集的方法;以及提取高分辨率网格、平面图和其他常见交付内容的方法。第二个行动计划是合并语义信息和解码器，以进行计数、测量和变化检测。这包括在连续嵌入中对语义进行编码，并为可视化、计数、测量和其他场景范围的几何语义查询创建解码器，以实现实时、灵活的映射和设施评估。第三个行动计划是在直接观测之外进行推断，并通过整合生成性和预测性过程，在新观测到达时推断和更新模型。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will investigate methods to create, from multiple images, a scene model that enables visualization, synthesis, counting, measurement, and other analysis. The goals of the project are driven by the need for unified geometric (where, what shape, how big) and semantic (what is it, what is it like) scene models, based on the investigators' direct experience in building products for construction management and vehicle safety. So far, computer vision has arguably had its largest impact in internet domains. This project is needed for broader applications involving the physical world, and the potential impact is hard to overstate. Resulting capabilities will lay foundations for real-time modeling, augmented reality, simulation, and robotics applications. The project lays the groundwork for a queryable, editable, and actionable semantic and geometric scene model, a foundational problem in computer vision. Neural rendering fields, vision language models, and diffusion have been impressively demonstrated for separate image synthesis and analysis applications. The project brings these advances together to enable new representations and capabilities for 3D semantic scene modeling. The result is a scalable and robust approach to create, update, query, and edit models of the world inferred from multiple observations. In particular, the project involves three plans of action. The first is to create measurable and meshable 3D scene models that can be efficiently estimated from sparse views and scale to thousands of images. This includes several developments: new efficiently optimizable, compact representations; incorporation of monocular geometry estimates; joint refinement of pose, gain, and other parameters; methods to scale seamlessly to massive scenes and photo sets; and ways to extract high resolution meshes, floor maps, and other common deliverables. The second plan of action is to incorporate semantic information and decoders for counting, measuring, and change detection. This includes encoding semantics in continuous embeddings and creating decoders for visualizing, counting, measuring, and other scene-wide geometric-semantic queries, to enable real-time, flexible mapping and facility assessment. The third plan of action is to extrapolate beyond direct observations and infer and update models as new observations arrive by integrating generative and predictive processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.