CRII: RI: RUI: Representations for multi-timescale scene dynamics in webcam video streams

数以千计的网络摄像头遍布世界各地，不断地对各种场景进行高清直播。由于它们的寿命长，这些流捕捉许多视觉上有趣的现象，不仅发生在真实的时间，但在更长的时间尺度。虽然计算机视觉算法已经被开发用于视频和场景理解，但这些网络摄像头流的数据量和时间的规模需要新的方法。该项目将开发表示，促进技术自动提取有关固定视图网络摄像头场景中的短期和长期变化的见解。这些技术可以应用于各种领域，包括安全，监视和各种监测用例，包括气候科学，生态学和发展。在以教学为导向的公立私立大学本科生将通过直接参与研究的各个方面获得宝贵的技能和经验。现有的视频理解和分析技术几乎只关注实时动态，如人类行为，而很少关注长期现象。这项研究的目的是开发表示长达数年的视频流，促进分析和理解的现象，在广泛的时间尺度。为了达到这个目的，表示需要紧凑，允许输入的完全重建，并按时间尺度组织场景内容。为了实现这一目标，将收集视频流的数据集，并训练场景特定的模型，以将视频帧编码为紧凑的潜在向量。一种新颖的正则化方案将在潜在空间中赋予秩序，根据场景内容的变化速度来安排场景内容的表示。这项研究将使新的方法来检测，监测和理解发生在数周，数月和数年的现象。现有的视频理解应用程序，如异常检测，动作识别和视频预测，也将受益于在紧凑和有组织的自我监督潜在空间中运行。最后，该研究将进一步了解如何在视频中编码时间冗余，并通过帮助确定在没有摄像机运动和大量观察的情况下可能实现的功能，对一般视频理解产生见解。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Thousands of webcams positioned all over the world are constantly streaming live HD views of a wide variety of scenes. Thanks to their longevity, these streams capture many visually interesting phenomena that occur not only in real time but over longer scales of time. Although computer vision algorithms have been developed for video and scene understanding, the scale - both in data volume and in time - of these webcam streams require new approaches. This project will develop representations that facilitate techniques to automatically extract insights about short and long-term changes in fixed-view webcam scenes. These techniques can be applied to a variety of areas including security, surveillance, and a diverse array of monitoring use cases including climate science, ecology, and development. Undergraduate students at a teaching-oriented public Primarily Undergraduate Institution will gain valuable skills and experience by being directly involved in all aspects of the research.Existing video understanding and analysis techniques focus almost exclusively on real-time dynamics, such as human actions, while little attention has been paid to longer-term phenomena. The objective of this research is to develop representations for years-long video streams that facilitate analysis and understanding of phenomena at a wide range of time scales. To serve this purpose, the representations need to be compact, allow for full reconstruction of the input, and organize scene content by timescale. To achieve this, a dataset of video streams will be collected and scene-specific models will be trained to encode video frames into compact latent vectors. A novel regularization scheme will impose order in the latent space, arranging the representation of the scene's content by how quickly it varies. The research will enable new ways to detect, monitor, and understand phenomena that occur over weeks, months, and years. Existing video understanding applications such as anomaly detection, action recognition, and video prediction will also benefit by operating in a compact and organized self-supervised latent space. Finally, the research will further progress towards understanding how to encode temporal redundancy in videos and yield insights about general video understanding by helping to determine what is possible with the absence of camera motion and an abundance of observations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.