CHS: Small: Integrative Wide-Area Augmented Reality Scene Modeling

增强现实（AR）是一种将计算机生成的视觉、听觉或其他感官信息叠加到物理世界上的技术，使它们看起来像是实际环境的一部分。该项目的目标是开发和评估用于创建、维护和改进大规模场景模型的新方法，以实现广域增强现实。例如，考虑一个针对消防员的AR应用程序，它叠加了火灾周边的预测变化和协调的行动计划，以帮助他们了解天气影响并传达当前的攻击计划。增强现实的特征在于，这些可视化与真实的世界三维配准，即使用户在其环境中移动，也能保持它们与物理环境实时匹配。为了放置新的虚拟注释（并跟踪先前放置的注释），需要创建表示物理世界位置的三维“场景模型”。理想情况下，这将在全球范围内完成，以包括所有可能的地方，其中AR体验是可能的，甚至已经发生。由于一个人或一个组织很难收集这种广域场景所需的数据，因此开发的系统将聚合不同模态的众包数据（例如，图像、视频和3D几何网格），利用除基本图像和点云数据之外的语义信息。通过提供远程引导本地AR用户捕获新图像或传感器数据以实现更准确或完整模型的功能，可以随着时间的推移引导众包建模以创建并不断更新大规模区域的场景模型，例如大学校园甚至城市。该研究将产生一个有效集成多个组件的系统，为政府、教育、工业和消费者领域的AR应用提供远程导航、探索和增强物理空间的新机会。为了实现上述目标，研究人员将实现并利用一个动态混合场景模型服务器，该服务器接受众包图像、视频和点云数据，并持续执行智能数据集成和完成，利用机器学习方法从原始图像和点云数据中推断语义信息并填充缺失信息。这方面的主要挑战将是设计学习方法，使其不需要访问所有低级别的数据融合和过滤组件，仅仅是因为信息可能无法在众包数据中获得。将开发和评估增强现实和虚拟现实用户界面，以处理服务器生成的不完美和不完整的环境模型，允许远程用户虚拟地浏览模型空间，并为本地AR用户提供指导。在人机界面方面，该项目侧重于远程导航和探索“视觉现实”的研究，从众包图像创建的真实世界空间的虚拟模型，以及创建增强视觉现实的内容。所提出的方法将解决当前混合现实应用在建模和远程导航方面的局限性，为用户提供远程视觉现实的体验，作为物理导航的替代品，作为计划活动的培训辅助工具，以及作为在增强和虚拟现实环境中共享信息的一种方式。该项目将利用研究人员团队和其他人在基于图像的建模，虚拟场景导航和远程协作虚拟注释方面的现有工作，该奖项反映了美国国家科学基金会的法定使命，并通过利用基金会的知识价值和更广泛的影响进行评估，被认为值得支持审查标准。

Augmented reality (AR) is a technology that superimposes computer-generated visual, auditory, or other sensory information onto the physical world, so that they appear to be part of the actual environment. The goal of the proposed project is to develop and evaluate new methods for creating, maintaining, and improving large-scale scene models to enable wide-area AR. For example, consider an AR application for firefighters, which superimposes predicted changes in fire perimeter and coordinated action plans to help them understand weather impact and communicate the current plan of attack. Augmented reality is characterized by the fact that these visualizations are three-dimensionally registered with the real world, keeping them matched to the physical environment in real-time even as the user moves around their environment. In order to place new virtual annotations (and to keep track of previously placed annotations), a three dimensional "scene model" that represents the physical world locations needs to be created. Ideally, this would be done on a global, world-wide scale to include all possible places where AR experiences are possible or have even already occurred. Since it is difficult for one person or organization to collect the data needed for such a wide-area scene, the developed system will aggregate crowd-sourced data of different modalities (e.g., images, videos, and 3D geometrical meshes) from multiple sources, leveraging semantic information in addition to basic image and point cloud data. By providing capabilities for remotely guiding a local AR user to capture new imagery or sensor data to achieve more accurate or complete models, crowd-sourced modeling can be directed over time to create and continuously update scene models of large-scale areas, such as a university campus or even a city. The research will result in a system that effectively integrates multiple components to provide new opportunities to remotely navigate, explore, and augment physical spaces for AR applications in government, education, industry, and consumer spaces.To accomplish the above objectives, the researchers will implement and utilize a Dynamic Hybrid Scene Model Server that accepts crowd-sourced image, video, and point cloud data, and continuously performs smart data integration and completion, leveraging machine learning approaches to infer semantic information from the raw image and point cloud data and to fill in missing information. The main challenge here will be to design the learning approaches in such a way that it will not require access to all the low-level data fusion and filtering components, simply because the information may not be available in the crowd-sourced data. Augmented Reality and Virtual Reality user interfaces will be developed and evaluated to deal with imperfect and incomplete environment models produced by the server, allowing remote users to virtually navigate through modeled spaces and to provide guidance to the local AR users. On the human interface side, the project focuses on research in remotely navigating and exploring "visual reality", virtual models of the real-world spaces created from the crowd-sourced imagery and creating content for augmenting the visual reality. The proposed methods will address limitations in current mixed reality applications in modeling and remote navigation, providing users an experience of remote visual reality that is valuable as a replacement for physical navigation, as a training aid for planned activity, and as a way to share information in augmented and virtual reality environments. The project will leverage existing work by the team of researchers and others on image-based modeling, virtual scene navigation, and virtual annotation for remote collaboration, and it focuses on both the key system components and the user experience to explicitly support navigation and augmentation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.