Structured Video Query Processing with Spatiotemporal Constraints

在智能手机和相机等拍摄设备普及的推动下，近几十年来，我们见证了视频数据的爆炸式增长。据思科称，到2022年，视频预计将占互联网流量的82%，每秒大约有120TB的视频数据通过互联网。这些视频包含了丰富的信息，可以用来提高我们的生产力、安全和生活质量。然而，由于我们目前查询和分析视频的方式所造成的限制，它们没有得到充分利用。另一方面，近年来深度学习（DL）的巨大进步已经彻底改变了许多具有重大实际意义的应用，包括目标检测和目标跟踪等计算机视觉任务。将现实世界的应用程序与DL算法和模型集成已经成为可能；通过应用专门的深度学习模型，可以理解对象类型（例如，人、车）及其在视频帧中的位置。然而，大多数现有的视频分析解决方案都是专门构建的，针对特定的应用；在研究和实践中，整合技术专业知识来构建和维护跨应用程序工作的视频分析所需的基础设施和平台仍然是极具挑战性的。拟议的研究计划旨在通过开发解决方案来解决这些挑战，这些解决方案弥合了强大的视频分析需求与特定深度学习模型能力之间的差距。研究计划的长期目标是为通用结构化视频处理和分析开发框架、模型、算法和系统，以释放（通过查询处理）视频数据的巨大潜力。由于视频中包含的时空信息是查询处理的关键，因此短期目标是开发用于处理视频数据结构化查询的时空数据结构、算法和模型。在以下研究问题上将取得进展：(1)如何从具有对象出现时间和方式约束的大型视频库中检索片段；(2)如何定量表示和存储一组对象之间不断变化的空间关系，并对这些关系进行高效、准确的查询处理；(3)如何基于视频片段的时空关联和视觉相似性对相同对象进行重新识别，以支持跨视频查询处理。所提出的研究将有助于为建立一个高度可扩展和能够执行细粒度时空模式匹配的通用视频分析系统奠定基础。它将帮助组织和个人用户从视频中发现有价值的信息，并在各种领域实现大量应用，如视频内容创建、执法、零售交通分析和自动驾驶。

Fueled by the prevalence of capturing devices such as smartphones and cameras, we have witnessed an explosion of video data over recent decades. According to Cisco, video is expected to make up 82% of Internet traffic by 2022, with approximately 120TB of video data crossing the Internet per second. Such videos contain a wealth of information to be tapped to benefit our productivity, safety, and quality of life. However, they are heavily under-utilized due to the limitations posed by our current way of querying and analyzing videos. On the other hand, the vast advances in Deep Learning (DL) in recent years have revolutionized numerous applications of major practical significance, including computer vision tasks such as object detection and object tracking. Integrating real-world applications with DL algorithms and models has become possible; understanding object types (e.g., person, car) and their locations in video frames is within reach by the application of specialized DL models. However, most existing video analytics solutions are purpose-built and target specific applications; assembling the technical expertise to build and maintain the required infrastructure and platform for video analytics that works across applications is still highly challenging, both in research and practice. The proposed research program aims to address these challenges by developing solutions that bridge the gap between the need for powerful video analytics and the capability of specific DL models. The long-term objective of the research program is to develop the frameworks, models, algorithms, and systems for general-purpose structured video processing and analytics to unleash (through query processing) the vast potential of video data. As spatiotemporal information contained in videos is key to the processing of queries, the short-term objective is to develop spatiotemporal data structures, algorithms, and models for the processing of structured queries over video data. Advances will be made to address the following research questions: (1) how to retrieve clips from large video repositories with constraints on when and how objects appear; (2) how to quantitatively represent and store the evolving spatial relationships among a group of objects and perform efficient and accurate query processing over such relationships; and (3) how to re-identify the same objects across video clips based on their spatiotemporal association and visual similarities to support cross-video query processing. The proposed research will help lay the foundation towards building a general-purpose video analytics system that is highly scalable and capable of performing fine-grained spatiotemporal pattern matching. It will help organizations and individual users uncover valuable information from videos and enable numerous applications in a wide variety of domains, such as video content creation, law enforcement, retail traffic analysis, and autonomous driving.