Declarative Query Processing Over Real Time Video Streams

点击翻译按钮获取中文摘要

In the last few years, Deep Learning (DL) has become a dominant artificial intelligence (AI) technology in industry and academia. It has managed to revolutionize certain important practical applications. Video data abound; as of this writing 500 hours of video are uploaded on Youtube every minute. Numerous applications benefit from advanced techniques to process and understand video content ranging from video surveillance and video monitoring applications, to news production and autonomous driving. State of the art DL algorithms assess the presence of specific objects in an image, assess their properties (e.g. color, texture), their location relative to the frame coordinates as well as track an object from frame to frame delivering impressive accuracy. In order to deliver solid results however, state of the art object detection techniques are far from real time and require significant hardware resources. Current technology, enables us to extract a schema from a video applying video classification/detection and tracking algorithms at the frame level. This presents an opportunity for data management. Our research explores declarative query processing on streaming video sources utilizing the schema extracted from video. Deep learning primitives (e.g., object detection) can be realized as User Defined Functions (UDF's) in a declarative framework. Our research explores the efficient identification of objects relevant to a query on the video streams, utilizing query specific networks (filters) that are built on demand, as opposed to large and more expensive deep learning models. Processing frames with cheaper query specific models, can quickly remove frames not relevant to the query and can dramatically improve frame processing rate. We explore options for applying such filters and explore optimization trade offs over speed and accuracy. Spatio-temporal query processing on detected objects is of vast importance. Queries can be expressed using spatial (e.g., human in-front-of car, etc) and/or temporal (e.g., car next-to stop-sign for 10 minutes) constraints in monitoring applications. At the same time interactions between objects (e.g., human breaking glass) are prevalent. This research studies how such predicates (spatial, temporal, actions, interactions) can be expressed and evaluated in a streaming video scenario when objects detected in frames are involved. We plan to support general conjunctive normal form queries on such predicates involving video objects. Last but not least, we extend the framework to support multiple video streams, as opposed to a single video stream. This raises numerous interesting research directions in correlating and matching objects across streams under spatial and temporal constraints and associated optimizations. This research is timely from a training perspective. It merges data management and deep learning primitives to build a real time video query processing system.