CAREER: Foundations of IoT-Supported Mobile Augmented Reality

移动增强现实将计算机生成的虚拟对象与用户周围的真实世界环境实时集成，有可能在物理世界的正确时间和正确地点为用户带来充满活力和身临其境的重要信息呈现。然而，现有的移动增强现实系统经常在识别真实世界中存在的对象时出错，从而为用户生成不正确的虚拟对象。他们还会在现实世界中放置虚拟物体时犯错误，使它们看起来像是移动、摆动或跳来跳去。该项目将通过利用位于移动增强现实设备附近的物联网(IoT)传感器和控制器来解决此类环境意识限制。许多物联网设备，如智能灯、摄像头和显示器，已经在智能家居、办公室和其他环境中普及。该项目将通过使用这些物联网设备作为有关真实世界的额外信息来源，并作为部分控制世界状态的手段，来提高移动增强现实环境意识。该项目将使移动增强现实更加可靠，为其在医疗和交通等关键健康和安全领域的应用铺平道路。该项目将在不同的研究生、本科生和高中生群体中培养多学科的研究技能。该项目的成果将在以K-12为导向的活动中作为互动演示展示，其特别目的是吸引妇女和代表不足的少数群体进入科学和工程专业。该项目将通过开发设计、实施和评估物联网支持的移动增强现实的技术，并通过演示物联网支持的方法来提高核心移动增强现实算法的性能，从而为物联网支持的移动增强现实研究奠定基础。本研究的目标分为三个方面。第一个推力将开发一个支持物联网的移动增强现实平台，一个基于游戏引擎的物联网支持的增强现实模拟器，以及新的物联网支持的移动增强现实应用。第二个推力将检查和量化移动增强现实语义感知方面的失败，并将开发通过利用物联网摄像头提供的同一物理空间的多个视图的可用性以及基于物联网的智能光影控制来解决这些失败的算法。第三个重点将检查和量化移动增强现实空间感知方面的故障，开发基于机器学习的技术来预测故障是否可能在给定环境中发生，并演示通过控制环境中可用的智能灯光、阴影和显示器的状态来改善空间感知的算法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile augmented reality, which integrates computer-generated virtual objects with real-world environments around a user in real time, has the potential to bring to the users, vibrant and immersive representations of important information at the right time and the right place within the physical world. Yet existing mobile augmented reality systems often make mistakes in identifying the objects present in the real world, thus generating incorrect virtual objects for the users. They also make mistakes in placing virtual objects within the real world, making them appear as shifting, oscillating, or jumping around. This project will address such environmental awareness limitations by taking advantage of Internet of Things (IoT) sensors and controllers located in the vicinity of mobile augmented reality devices. Many IoT devices, such as smart lights, cameras, and displays, are already pervasive in smart homes, offices, and other environments. This project will improve mobile augmented reality environmental awareness by using these IoT devices as sources of additional information about the real world, and as means for partially controlling the state of the world. The project will make mobile augmented reality more reliable, paving way for its applications in health-critical and safety-critical domains, such as medicine and transportation. The project will develop multi-disciplinary research skills in a diverse cohort of graduate, undergraduate, and high school students. The findings of this project will be showcased as interactive demonstrations at K-12-oriented events, with the particular aim of attracting women and underrepresented minorities to science and engineering. This project will lay the foundations for studies of IoT-supported mobile augmented reality by developing techniques for designing, implementing, and evaluating it, and by demonstrating IoT-supported methods that improve the performance of core mobile augmented reality algorithms. The goals of this research are divided into three thrusts. The first thrust will develop an IoT-supported mobile augmented reality platform, a game engine-based IoT-supported augmented reality emulator, and new IoT-supported mobile augmented reality applications. The second thrust will examine and quantify the failures in mobile augmented reality semantic awareness, and will develop algorithms that address these failures by making use of the IoT camera-provided availability of multiple views of the same physical space and IoT-based smart light and shade controls. The third thrust will examine and quantify failures in mobile augmented reality spatial awareness, develop machine learning-based techniques to predict whether the failures are likely to occur in a given environment, and demonstrate algorithms that improve spatial awareness by controlling the state of smart lights, shades, and displays available in the environment.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.