Structured Video Query Processing with Spatiotemporal Constraints

点击翻译按钮获取中文摘要

Fueled by the prevalence of capturing devices such as smartphones and cameras, we have witnessed an explosion of video data over recent decades. According to Cisco, video is expected to make up 82% of Internet traffic by 2022, with approximately 120TB of video data crossing the Internet per second. Such videos contain a wealth of information to be tapped to benefit our productivity, safety, and quality of life. However, they are heavily under-utilized due to the limitations posed by our current way of querying and analyzing videos. On the other hand, the vast advances in Deep Learning (DL) in recent years have revolutionized numerous applications of major practical significance, including computer vision tasks such as object detection and object tracking. Integrating real-world applications with DL algorithms and models has become possible; understanding object types (e.g., person, car) and their locations in video frames is within reach by the application of specialized DL models. However, most existing video analytics solutions are purpose-built and target specific applications; assembling the technical expertise to build and maintain the required infrastructure and platform for video analytics that works across applications is still highly challenging, both in research and practice. The proposed research program aims to address these challenges by developing solutions that bridge the gap between the need for powerful video analytics and the capability of specific DL models. The long-term objective of the research program is to develop the frameworks, models, algorithms, and systems for general-purpose structured video processing and analytics to unleash (through query processing) the vast potential of video data. As spatiotemporal information contained in videos is key to the processing of queries, the short-term objective is to develop spatiotemporal data structures, algorithms, and models for the processing of structured queries over video data. Advances will be made to address the following research questions: (1) how to retrieve clips from large video repositories with constraints on when and how objects appear; (2) how to quantitatively represent and store the evolving spatial relationships among a group of objects and perform efficient and accurate query processing over such relationships; and (3) how to re-identify the same objects across video clips based on their spatiotemporal association and visual similarities to support cross-video query processing. The proposed research will help lay the foundation towards building a general-purpose video analytics system that is highly scalable and capable of performing fine-grained spatiotemporal pattern matching. It will help organizations and individual users uncover valuable information from videos and enable numerous applications in a wide variety of domains, such as video content creation, law enforcement, retail traffic analysis, and autonomous driving.