CAREER: Machine-centered Cyberinfrastructure for Panoramic Video Analytics in Science and Engineering Monitoring

视频分析在科学和工程监控中起着举足轻重的作用。由于摄像机设备的计算能力有限，远程摄像机捕获的监控视频通常实时传输到服务器进行分析。从野生动物追踪和海岸线事件探测，到机场嫌疑人识别和灾难应对中的受害者搜索，此类自动视频分析系统已广泛部署，以协助操作员。360度摄像机的最新进展为全景视频分析提供了一个新的范例，它可以覆盖监测站点的360度环境，并可以解决传统2D视频分析中的错误和缺失的分析能力。然而，实现这一愿景需要将大量全景视频数据实时传输到服务器进行在线分析，这是当前网络基础设施（CI）无法支持的。360度视频比特率和可用网络带宽之间的不匹配会导致分析滞后或失败，从而降低全景视频分析的好处。该项目将创建一个视频压缩、流媒体和恢复的框架，以实现科学和工程监控中全景视频分析的愿景。新的CI将允许科学家和工程师进行在线全景视频分析，并使创新应用成为可能，否则无法实现。研究成果将支持影像分析的远程学习工具、媒体计算的课程设置和本科研究，以及用于公众宣传的教育视频的开发。本计画研究以机器为中心的视讯计算架构，以实现线上全景视讯分析。与传统的以人为中心的视频框架（处理像素以保留人类观看的广泛美学细节）不同，所提出的CI压缩、流式传输和恢复特征点以供机器分析。由于这一根本性的变化，所提出的框架能够大大优于传统的视频ci并支持全景视频分析。为此，将构建基于深度学习的360度视频编解码器，提取视频特征的时空特征，优化压缩比和分析精度。其次，将设计一个自适应360度视频比特率流系统，通过优先考虑机器感兴趣的区域，确保连续传输完整的360度视频帧。第三，将开发360度视频恢复方案，在考虑在线分析模型的时间限制的情况下，恢复噪声和延迟的视频数据。最后，将与应用领域的科学家和工程师进行跨学科合作，开展项目计划，以评估和验证全景视频框架对现实世界问题的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Video analytics plays a pivotal role in science and engineering monitoring. Monitoring videos captured by remote cameras are typically live streamed to servers for analysis because of the limited computational capabilities of camera devices. From wildlife tracking and coastline event detection to airport suspect recognition and victim search in disaster response, such automated video analytics systems have been deployed widely to assist human operators. The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable. The research outcomes will support the development of a remote learning tool for imaging analytics, course curriculum and undergraduate research in media computing, and educational videos for public outreach.This project investigates a machine centered video computing framework in order to enable online panoramic video analytics. Unlike traditional human centered video frameworks where pixels are processed to preserve extensive aesthetic details for human viewing, the proposed CI compresses, streams, and recovers feature points for machine analytics. Because of this fundamental change, the proposed framework is able to greatly outperform legacy video CIs and support panoramic video analytics. To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics models. Finally, interdisciplinary collaboration will be done with application area scientists and engineers to carry out the project plans for evaluation and validation of the panoramic video framework on real world problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.