FW-HTF: First Person View and Augmented Reality for Airborne Embodied Intelligent Cognitive Assistants

人类技术前沿工作的未来（FW-HTF）是美国国家科学基金会（NSF）宣布的未来投资十大新构想之一。FW-HTF跨部门计划旨在通过支持融合研究来应对不断变化的就业和工作环境的挑战和机遇。这个奖项部分实现了这一目标。未来的工作将涉及人类操作员和半自动机器人系统。人类工人控制机器人，扩展它们的感知和物理能力。这些由技术驱动的劳动力将从事远程、危险和日益专业化的工作。这项工作的一个例子可能是检查难以接近的桥梁支撑。该项目将探索使用智能机载无人机来帮助地面操作员检查公路桥梁。通过将增强现实（AR）与第一人称视角（FPV）操作员界面仔细整合，本研究将提高工作人员在各种其他非常困难的任务中的表现。该研究计划将解决在基础设施检查中使用嵌入式智能认知助手（e-ICAs）的关键挑战。将通过AR/FPV用户界面为人/机器人协作开发共享态势感知的新原则，这些原则将为界面设计指南和评估措施提供信息。该研究项目还将强调协同感知和规划，例如增强共享态势感知的外围/中央计算机视觉，改善图像质量的注视信息自适应视点规划，以及针对部分已知和不确定地图的全局规划方法，当工作人员发现新信息时实时调整计划。可调控制系统性能将允许工作人员和e-ICA协作管理干扰能量，以优化任务数据质量和飞行续航力，同时确保飞行安全。包含带注释的退化图像的公共存储库的并行开发将支持基于人工智能的缺陷分析，从而为人类工作人员提供实时检查线索。一项平行和协调的经济和劳动力分析将评估机载e- ica的影响，使用FPV和增强现实控制，对未来基础设施检查等工作的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The Future of Work at the Human-Technology Frontier (FW-HTF) is one of 10 new Big Ideas for Future Investment announced by NSF. The FW-HTF cross-directorate program aims to respond to the challenges and opportunities of the changing landscape of jobs and work by supporting convergent research. This award fulfills part of that aim. The future of work will involve human operators and semi-autonomous robotic systems. Human workers control the robots, extending their sensing and physical capabilities. This technology-empowered workforce will do remote, dangerous, and increasingly specialized work. An example use of this work might be the inspection of hard-to-access bridge supports. This project will explore the use of intelligent airborne drones to help ground-based operators in the inspection of highway bridges. Through the careful integration of augmented reality (AR) with first-person-view (FPV) operator interfaces this research will enhance worker performance across a wide range of otherwise very difficult tasks. The research program will address key challenges in the use of embodied intelligent cognitive assistants (e-ICAs) for infrastructure inspection. New principles of shared situation awareness will be developed for human/robot collaboration through AR/FPV user interfaces and these principles will inform interface design guidelines and evaluation measures. The research program will also emphasize collaborative perception and planning, such as peripheral/central computer vision to enhance shared situation awareness, gaze-informed adaptive viewpoint planning for improved image quality, and a global planning method for partially known and uncertain maps that adapts the plan in real-time as the worker discovers new information. Tunable control system performance will allow the worker and the e-ICA to collaborate in managing disturbance energy to optimize mission data quality and flight endurance while ensuring safety of flight. The parallel development of a public repository containing annotated deterioration imagery will support artificial intelligence based defect analytics to provide the human worker with real-time inspection cues. A parallel and coordinated economic and workforce analysis will assess the impact of airborne e-ICAs, controlled using FPV with augmented reality, on the future of work in infrastructure inspection and beyond.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.