CHS: Small: Integrative Wide-Area Augmented Reality Scene Modeling

增强现实(AR)是一种将计算机生成的视觉、听觉或其他感官信息叠加到物理世界上的技术，使它们看起来像是实际环境的一部分。拟议项目的目标是开发和评估创建、维护和改进大规模场景模型的新方法，以实现广域增强现实。例如，考虑一个针对消防员的AR应用程序，该应用程序叠加了消防周长的预测变化和协调的行动计划，以帮助他们了解天气影响并传达当前的攻击计划。增强现实的特点是，这些可视化是与真实世界三维注册的，即使用户在他们的环境中移动，也能保持它们与物理环境实时匹配。为了放置新的虚拟注释(并跟踪先前放置的注释)，需要创建表示物理世界位置的三维“场景模型”。理想情况下，这将在全球、世界范围内进行，包括所有可能发生或甚至已经发生AR体验的地方。由于一个人或组织很难收集这种广域场景所需的数据，因此开发的系统将从多个来源聚合不同形态(例如，图像、视频和3D几何网格)的众包数据，除了基本的图像和点云数据外，还利用语义信息。通过提供远程引导本地AR用户捕获新图像或传感器数据的功能，以实现更准确或更完整的模型，可以随着时间的推移指导众包建模，以创建并持续更新大规模区域的场景模型，例如大学校园甚至城市。为了实现上述目标，研究人员将实现并利用动态混合场景模型服务器，该服务器接受来自众包的图像、视频和点云数据，并持续执行智能数据集成和完成，利用机器学习方法从原始图像和点云数据中推断语义信息并填充缺失信息。这里的主要挑战将是设计学习方法，使其不需要访问所有低级数据融合和过滤组件，仅仅是因为众包数据中可能没有这些信息。将开发和评估增强现实和虚拟现实用户界面，以处理服务器产生的不完美和不完整的环境模型，使远程用户能够在建模空间中虚拟导航，并为本地AR用户提供指导。在人机界面方面，该项目的重点是远程导航和探索“视觉现实”，即从众包图像中创建的现实世界空间的虚拟模型，并创建用于增强视觉现实的内容。拟议的方法将解决目前混合现实应用在建模和远程导航方面的局限性，为用户提供远程视觉现实体验，作为物理导航的替代、作为计划活动的培训辅助以及作为在增强现实和虚拟现实环境中共享信息的一种方式。该项目将利用研究人员团队和其他人在基于图像的建模、虚拟场景导航和用于远程协作的虚拟注释方面的现有工作，并将重点放在关键系统组件和用户体验上，以明确支持导航和增强。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) is a technology that superimposes computer-generated visual, auditory, or other sensory information onto the physical world, so that they appear to be part of the actual environment. The goal of the proposed project is to develop and evaluate new methods for creating, maintaining, and improving large-scale scene models to enable wide-area AR. For example, consider an AR application for firefighters, which superimposes predicted changes in fire perimeter and coordinated action plans to help them understand weather impact and communicate the current plan of attack. Augmented reality is characterized by the fact that these visualizations are three-dimensionally registered with the real world, keeping them matched to the physical environment in real-time even as the user moves around their environment. In order to place new virtual annotations (and to keep track of previously placed annotations), a three dimensional "scene model" that represents the physical world locations needs to be created. Ideally, this would be done on a global, world-wide scale to include all possible places where AR experiences are possible or have even already occurred. Since it is difficult for one person or organization to collect the data needed for such a wide-area scene, the developed system will aggregate crowd-sourced data of different modalities (e.g., images, videos, and 3D geometrical meshes) from multiple sources, leveraging semantic information in addition to basic image and point cloud data. By providing capabilities for remotely guiding a local AR user to capture new imagery or sensor data to achieve more accurate or complete models, crowd-sourced modeling can be directed over time to create and continuously update scene models of large-scale areas, such as a university campus or even a city. The research will result in a system that effectively integrates multiple components to provide new opportunities to remotely navigate, explore, and augment physical spaces for AR applications in government, education, industry, and consumer spaces.To accomplish the above objectives, the researchers will implement and utilize a Dynamic Hybrid Scene Model Server that accepts crowd-sourced image, video, and point cloud data, and continuously performs smart data integration and completion, leveraging machine learning approaches to infer semantic information from the raw image and point cloud data and to fill in missing information. The main challenge here will be to design the learning approaches in such a way that it will not require access to all the low-level data fusion and filtering components, simply because the information may not be available in the crowd-sourced data. Augmented Reality and Virtual Reality user interfaces will be developed and evaluated to deal with imperfect and incomplete environment models produced by the server, allowing remote users to virtually navigate through modeled spaces and to provide guidance to the local AR users. On the human interface side, the project focuses on research in remotely navigating and exploring "visual reality", virtual models of the real-world spaces created from the crowd-sourced imagery and creating content for augmenting the visual reality. The proposed methods will address limitations in current mixed reality applications in modeling and remote navigation, providing users an experience of remote visual reality that is valuable as a replacement for physical navigation, as a training aid for planned activity, and as a way to share information in augmented and virtual reality environments. The project will leverage existing work by the team of researchers and others on image-based modeling, virtual scene navigation, and virtual annotation for remote collaboration, and it focuses on both the key system components and the user experience to explicitly support navigation and augmentation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.