CHS: Small: Integrative Wide-Area Augmented Reality Scene Modeling

增强现实（AR）是一种将计算机生成的视觉、听觉或其他感官信息叠加到物理世界上的技术，使它们看起来像是实际环境的一部分。拟议项目的目标是开发和评估用于创建、维护和改进大规模场景模型的新方法，以实现广域增强现实。例如，考虑消防员的增强现实应用程序，该应用程序可以叠加火灾范围的预测变化和协调行动计划，以帮助他们了解天气影响并传达当前的攻击计划。增强现实的特点是，这些可视化与现实世界进行了三维注册，即使用户在他们的环境中移动，也能实时保持它们与物理环境相匹配。为了放置新的虚拟注释（并跟踪先前放置的注释），需要创建一个代表物理世界位置的三维“场景模型”。理想情况下，这将在全球范围内完成，包括所有可能实现或已经实现AR体验的地方。由于一个人或组织很难收集到如此广域场景所需的数据，因此开发的系统将聚合来自多个来源的不同模式的众包数据（例如，图像，视频和3D几何网格），除了利用基本图像和点云数据外，还利用语义信息。通过提供远程引导本地AR用户捕获新图像或传感器数据以获得更准确或更完整的模型的功能，众包建模可以随着时间的推移而创建并不断更新大规模区域的场景模型，例如大学校园甚至城市。该研究将产生一个有效集成多个组件的系统，为AR应用在政府、教育、工业和消费者空间中的远程导航、探索和增强物理空间提供新的机会。为了实现上述目标，研究人员将实施和利用一个动态混合场景模型服务器，该服务器接受众包图像、视频和点云数据，并不断执行智能数据集成和完成，利用机器学习方法从原始图像和点云数据中推断语义信息，并填补缺失信息。这里的主要挑战将是设计学习方法，使其不需要访问所有底层数据融合和过滤组件，因为信息可能在众包数据中不可用。增强现实和虚拟现实用户界面将被开发和评估，以处理由服务器产生的不完善和不完整的环境模型，允许远程用户虚拟地在建模空间中导航，并为本地AR用户提供指导。在人机界面方面，该项目侧重于研究远程导航和探索“视觉现实”，从众包图像中创建现实世界空间的虚拟模型，并创建增强视觉现实的内容。所提出的方法将解决当前混合现实应用在建模和远程导航方面的局限性，为用户提供远程视觉现实体验，作为物理导航的替代品，作为计划活动的培训辅助，以及在增强和虚拟现实环境中共享信息的一种方式。该项目将利用研究团队和其他人员在基于图像的建模、虚拟场景导航和远程协作的虚拟注释方面的现有工作，并将重点放在关键系统组件和用户体验上，以明确支持导航和增强。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) is a technology that superimposes computer-generated visual, auditory, or other sensory information onto the physical world, so that they appear to be part of the actual environment. The goal of the proposed project is to develop and evaluate new methods for creating, maintaining, and improving large-scale scene models to enable wide-area AR. For example, consider an AR application for firefighters, which superimposes predicted changes in fire perimeter and coordinated action plans to help them understand weather impact and communicate the current plan of attack. Augmented reality is characterized by the fact that these visualizations are three-dimensionally registered with the real world, keeping them matched to the physical environment in real-time even as the user moves around their environment. In order to place new virtual annotations (and to keep track of previously placed annotations), a three dimensional "scene model" that represents the physical world locations needs to be created. Ideally, this would be done on a global, world-wide scale to include all possible places where AR experiences are possible or have even already occurred. Since it is difficult for one person or organization to collect the data needed for such a wide-area scene, the developed system will aggregate crowd-sourced data of different modalities (e.g., images, videos, and 3D geometrical meshes) from multiple sources, leveraging semantic information in addition to basic image and point cloud data. By providing capabilities for remotely guiding a local AR user to capture new imagery or sensor data to achieve more accurate or complete models, crowd-sourced modeling can be directed over time to create and continuously update scene models of large-scale areas, such as a university campus or even a city. The research will result in a system that effectively integrates multiple components to provide new opportunities to remotely navigate, explore, and augment physical spaces for AR applications in government, education, industry, and consumer spaces.To accomplish the above objectives, the researchers will implement and utilize a Dynamic Hybrid Scene Model Server that accepts crowd-sourced image, video, and point cloud data, and continuously performs smart data integration and completion, leveraging machine learning approaches to infer semantic information from the raw image and point cloud data and to fill in missing information. The main challenge here will be to design the learning approaches in such a way that it will not require access to all the low-level data fusion and filtering components, simply because the information may not be available in the crowd-sourced data. Augmented Reality and Virtual Reality user interfaces will be developed and evaluated to deal with imperfect and incomplete environment models produced by the server, allowing remote users to virtually navigate through modeled spaces and to provide guidance to the local AR users. On the human interface side, the project focuses on research in remotely navigating and exploring "visual reality", virtual models of the real-world spaces created from the crowd-sourced imagery and creating content for augmenting the visual reality. The proposed methods will address limitations in current mixed reality applications in modeling and remote navigation, providing users an experience of remote visual reality that is valuable as a replacement for physical navigation, as a training aid for planned activity, and as a way to share information in augmented and virtual reality environments. The project will leverage existing work by the team of researchers and others on image-based modeling, virtual scene navigation, and virtual annotation for remote collaboration, and it focuses on both the key system components and the user experience to explicitly support navigation and augmentation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.