CAREER: Foundations of IoT-Supported Mobile Augmented Reality

移动增强现实技术将计算机生成的虚拟对象与用户周围的真实世界环境实时集成在一起，有可能在物理世界的正确时间和正确地点为用户带来重要信息的生动而身临其境的表现。然而，现有的移动增强现实系统在识别现实世界中的物体时经常出现错误，从而为用户生成不正确的虚拟物体。他们也会在将虚拟物体放置在现实世界中时犯错误，使它们看起来像是在移动、振荡或跳跃。该项目将利用位于移动增强现实设备附近的物联网（IoT）传感器和控制器来解决这种环境意识的局限性。许多物联网设备，如智能灯、摄像头和显示器，已经在智能家居、办公室和其他环境中普及。该项目将通过使用这些物联网设备作为关于现实世界的额外信息来源，以及作为部分控制世界状态的手段，提高移动增强现实环境意识。该项目将使移动增强现实更加可靠，为其在医疗和运输等健康和安全关键领域的应用铺平道路。该项目将培养研究生、本科生和高中生的多学科研究技能。该项目的研究结果将在面向k -12的活动中以互动演示的形式展示，其特别目的是吸引妇女和代表性不足的少数民族进入科学和工程领域。该项目将通过开发设计、实施和评估技术，并通过展示提高核心移动增强现实算法性能的物联网支持方法，为物联网支持的移动增强现实研究奠定基础。本研究的目标分为三个重点。第一个重点将开发一个支持物联网的移动增强现实平台，一个基于游戏引擎的支持物联网的增强现实模拟器，以及新的支持物联网的移动增强现实应用程序。第二个重点将检查和量化移动增强现实语义感知中的故障，并将开发算法，通过利用物联网摄像头提供的相同物理空间的多个视图的可用性和基于物联网的智能光影控制来解决这些故障。第三个重点将检查和量化移动增强现实空间感知中的故障，开发基于机器学习的技术来预测故障是否可能在给定环境中发生，并演示通过控制环境中可用的智能灯光、阴影和显示器的状态来提高空间感知的算法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile augmented reality, which integrates computer-generated virtual objects with real-world environments around a user in real time, has the potential to bring to the users, vibrant and immersive representations of important information at the right time and the right place within the physical world. Yet existing mobile augmented reality systems often make mistakes in identifying the objects present in the real world, thus generating incorrect virtual objects for the users. They also make mistakes in placing virtual objects within the real world, making them appear as shifting, oscillating, or jumping around. This project will address such environmental awareness limitations by taking advantage of Internet of Things (IoT) sensors and controllers located in the vicinity of mobile augmented reality devices. Many IoT devices, such as smart lights, cameras, and displays, are already pervasive in smart homes, offices, and other environments. This project will improve mobile augmented reality environmental awareness by using these IoT devices as sources of additional information about the real world, and as means for partially controlling the state of the world. The project will make mobile augmented reality more reliable, paving way for its applications in health-critical and safety-critical domains, such as medicine and transportation. The project will develop multi-disciplinary research skills in a diverse cohort of graduate, undergraduate, and high school students. The findings of this project will be showcased as interactive demonstrations at K-12-oriented events, with the particular aim of attracting women and underrepresented minorities to science and engineering. This project will lay the foundations for studies of IoT-supported mobile augmented reality by developing techniques for designing, implementing, and evaluating it, and by demonstrating IoT-supported methods that improve the performance of core mobile augmented reality algorithms. The goals of this research are divided into three thrusts. The first thrust will develop an IoT-supported mobile augmented reality platform, a game engine-based IoT-supported augmented reality emulator, and new IoT-supported mobile augmented reality applications. The second thrust will examine and quantify the failures in mobile augmented reality semantic awareness, and will develop algorithms that address these failures by making use of the IoT camera-provided availability of multiple views of the same physical space and IoT-based smart light and shade controls. The third thrust will examine and quantify failures in mobile augmented reality spatial awareness, develop machine learning-based techniques to predict whether the failures are likely to occur in a given environment, and demonstrate algorithms that improve spatial awareness by controlling the state of smart lights, shades, and displays available in the environment.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.