EAGER: Collaborative Research: Augmented 360 Video for Situation Awareness in Firefighting

火灾事故造成了消防员和平民的严重受伤、疾病和死亡。控制中心的指挥官和在紧急现场工作的消防员之间的沟通不畅经常被认为是消防员行动的死亡和损失报告的决定因素。传统上，远程指挥官可视化紧急现场，并使用消防员头盔摄像头捕获的视频领导行动。不幸的是，这些视频系统存在一个基本问题，即，指挥官一次只能看到紧急现场的一个视图。这种限制限制了指挥官的态势感知能力，并导致生产力和安全问题，例如位置的错误传达和无法识别危险事件。该项目结合了360度视频和增强现实技术，使远程指挥官能够在所有观看方向上实现整个紧急现场的360度态势感知，并提高消防生产力和安全性。360度视频观看有利于各种应急响应社区的规划和培训。其他研究成果，包括开放数据集和软件，通过出版物，演示文稿和网站广泛传播，为计算机科学和工程界做出贡献。 还计划开展教育活动，包括本科生研究以及K-12学生的消防安全培训，以加强该项目的影响。 该项目将设计和开发增强的360度视频技术，使远程指挥官能够在360度场景内切换视口，并可视化机器检测到的感兴趣的事件。本研究将探讨一种增强的360度视频观看系统，以通过以下步骤实现消防行动中事件指挥的全景态势感知：（1）针对指挥官对消防视频中态势感知的需求和要求，将进行访谈以识别和分类消防响应中的重要对象和事件及其技术偏好;（2）针对消防员视频中重要事件的自动机器检测的知识空白，设计了一个兴趣视图检测模型，检测目标对象和事件;（3）为指挥员提供全景态势感知，美国国家科学基金会将开发一种用于远程事故指挥的增强型360度视频观看系统，并通过实地研究对其进行评估。该奖项反映了美国国家科学基金会的法定使命，并通过利用基金会的智力价值和更广泛的知识进行评估，被认为是值得支持的影响审查标准。

Fire incidents have caused substantial injuries, illness, and death on both firefighters and civilians. Poor communications between commanders in the control center and firefighters working at emergency sites are frequently cited as the determining factor in fatality and loss reports of firefighter operations. Traditionally, remote commanders visualize emergency sites and lead the operation using videos captured by helmet cameras of firefighters. Unfortunately, these video systems suffer from a fundamental problem, i.e., commanders are only able to see a single view of the emergency site at a time. This limitation restricts the situation awareness of commanders and leads to productivity and safety issues such as miscommunication of locations and failure to identify dangerous events. This project combines 360-degree videos and augmented reality to enable remote commanders to achieve 360-degree situation awareness of the entire emergency site in all viewing directions and enhance firefighting productivity and safety. The 360-degree video viewing benefits various emergency response communities in planning and training. Other research outcomes, including open datasets and software, are widely disseminated through publications, presentations, and websites to contribute to the computer science and engineering communities. Educational activities including undergraduate research as well as fire safety training for K-12 students are also planned to enhance the impacts of this project. This project will design and develop augmented 360 video technology to enable remote commanders to switch viewports within a 360-degree scene and visualize machine-detected events of interest. It will investigate an augmented 360 video viewing system to enable panoramic situation awareness for incident command in firefighter operations through these steps: (1) to address commanders’ needs and requirements for situation awareness in firefighter videos, interviews will be performed to identify and categorize important objects and events in firefighter response and their technology preferences; (2) to fill in the knowledge gap of how to enable automatic machine detection of important events in firefighter videos, a view of interest detection model will be designed detects target objects and events; and (3) to provide panoramic situation awareness to commanders, an augmented 360 video viewing system for remote incident command will be developed and evaluated through a field study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.