CAREER: Machine-centered Cyberinfrastructure for Panoramic Video Analytics in Science and Engineering Monitoring

视频分析在科学和工程监控中发挥着关键作用。由于摄像机设备的计算能力有限，由远程摄像机捕获的监控视频通常被实时流传输到服务器以供分析。从野生动物跟踪和海岸线事件检测到机场嫌疑人识别和灾难响应中的受害者搜索，这种自动化视频分析系统已被广泛部署，以协助人类操作员。360度摄像机的最新进展实现了全景视频分析的新范式，可以覆盖监控现场的360度环境，并可以解决传统2D视频分析中的错误和缺失的分析能力。然而，实现这一愿景需要将大量全景视频数据实时流式传输到服务器进行在线分析，而当前的网络基础设施（CI）无法支持这一点。360度视频比特率与可用网络带宽之间的不匹配可能导致分析滞后或失败，从而降低全景视频分析的优势。该项目将创建一个视频压缩，流媒体和恢复的框架，以实现科学和工程监控中全景视频分析的愿景。新的CI将允许科学家和工程师进行在线全景视频分析，并实现其他方式无法实现的创新应用。研究成果将支持开发用于图像分析的远程学习工具，媒体计算课程和本科生研究，以及用于公共宣传的教育视频。该项目研究了以机器为中心的视频计算框架，以实现在线全景视频分析。与传统的以人为中心的视频框架不同，在传统的视频框架中，像素被处理以保留大量的美学细节供人类观看，所提出的CI压缩、流式传输和恢复特征点以供机器分析。由于这种根本性的变化，所提出的框架能够大大优于传统的视频CI，并支持全景视频分析。为此，将构建基于深度学习的360度视频编解码器，以提取视频特征的时空特征，并优化压缩比和分析准确性。其次，将设计一个自适应360度视频比特率流媒体系统，通过优先考虑机器首选的感兴趣区域，确保连续传输完整的360度视频帧。第三，将开发360度视频恢复方案，以恢复有噪声和延迟的视频数据，同时考虑在线分析模型中的时间限制。最后，将与应用领域的科学家和工程师进行跨学科合作，以执行关于真实的世界问题的全景视频框架评估和验证的项目计划。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Video analytics plays a pivotal role in science and engineering monitoring. Monitoring videos captured by remote cameras are typically live streamed to servers for analysis because of the limited computational capabilities of camera devices. From wildlife tracking and coastline event detection to airport suspect recognition and victim search in disaster response, such automated video analytics systems have been deployed widely to assist human operators. The recent advancement of 360 degree cameras enables a new paradigm of panoramic video analytics that can cover the 360 degree surroundings of a monitoring site and can address the errors in and missing analysis abilities of traditional 2D video analytics. However, realizing this vision requires live streaming massive panoramic video data to servers for online analytics, which cannot be supported by the current cyberinfrastructure (CI). The mismatch between the 360 degree video bit rate and available network bandwidth can cause lagging or failed analysis, diminishing the benefits of panoramic video analytics. This project will create a framework of video compression, streaming, and recovery for achieving the vision of panoramic video analytics in science and engineering monitoring. The new CI will allow scientists and engineers to conduct online panoramic video analytics and enable innovative applications that are otherwise unattainable. The research outcomes will support the development of a remote learning tool for imaging analytics, course curriculum and undergraduate research in media computing, and educational videos for public outreach.This project investigates a machine centered video computing framework in order to enable online panoramic video analytics. Unlike traditional human centered video frameworks where pixels are processed to preserve extensive aesthetic details for human viewing, the proposed CI compresses, streams, and recovers feature points for machine analytics. Because of this fundamental change, the proposed framework is able to greatly outperform legacy video CIs and support panoramic video analytics. To this end, a deep learning based 360 degree video codec will be built to distill the spatiotemporal characteristics of video features and optimize both compression ratio and analytics accuracy. Second, an adaptive 360 degree video bitrate streaming system will be designed to ensure continuous delivery of full 360 degree video frames by prioritizing regions of interest preferred by machines. Third, a 360 degree video recovery scheme will be developed to restore noisy and delayed video data while considering the time constraints in the online analytics models. Finally, interdisciplinary collaboration will be done with application area scientists and engineers to carry out the project plans for evaluation and validation of the panoramic video framework on real world problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.