Declarative Query Processing Over Real Time Video Streams

实时视频流上的声明式查询处理

• 批准号: RGPIN-2020-07238
• 负责人: Koudas, Nikolaos
• 金额: $ 2.11万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=749627
• 关键词: Declarative; Query; Processing; Over; Real

In the last few years, Deep Learning (DL) has become a dominant artificial intelligence (AI) technology in industry and academia. It has managed to revolutionize certain important practical applications. Video data abound; as of this writing 500 hours of video are uploaded on Youtube every minute. Numerous applications benefit from advanced techniques to process and understand video content ranging from video surveillance and video monitoring applications, to news production and autonomous driving. State of the art DL algorithms assess the presence of specific objects in an image, assess their properties (e.g. color, texture), their location relative to the frame coordinates as well as track an object from frame to frame delivering impressive accuracy. In order to deliver solid results however, state of the art object detection techniques are far from real time and require significant hardware resources. Current technology, enables us to extract a schema from a video applying video classification/detection and tracking algorithms at the frame level. This presents an opportunity for data management. Our research explores declarative query processing on streaming video sources utilizing the schema extracted from video. Deep learning primitives (e.g., object detection) can be realized as User Defined Functions (UDF's) in a declarative framework. Our research explores the efficient identification of objects relevant to a query on the video streams, utilizing query specific networks (filters) that are built on demand, as opposed to large and more expensive deep learning models. Processing frames with cheaper query specific models, can quickly remove frames not relevant to the query and can dramatically improve frame processing rate. We explore options for applying such filters and explore optimization trade offs over speed and accuracy. Spatio-temporal query processing on detected objects is of vast importance. Queries can be expressed using spatial (e.g., human in-front-of car, etc) and/or temporal (e.g., car next-to stop-sign for 10 minutes) constraints in monitoring applications. At the same time interactions between objects (e.g., human breaking glass) are prevalent. This research studies how such predicates (spatial, temporal, actions, interactions) can be expressed and evaluated in a streaming video scenario when objects detected in frames are involved. We plan to support general conjunctive normal form queries on such predicates involving video objects. Last but not least, we extend the framework to support multiple video streams, as opposed to a single video stream. This raises numerous interesting research directions in correlating and matching objects across streams under spatial and temporal constraints and associated optimizations. This research is  timely from a training perspective. It merges data management and deep learning primitives to build a real time video query processing system.

在过去的几年里，深度学习（DL）已经成为工业界和学术界占主导地位的人工智能（AI）技术。它成功地革新了某些重要的实际应用。视频数据丰富；在撰写本文时，每分钟有500个小时的视频上传到Youtube。从视频监控和视频监控应用到新闻制作和自动驾驶，许多应用都受益于先进的技术来处理和理解视频内容。最先进的深度学习算法评估图像中特定物体的存在，评估它们的属性（例如颜色，纹理），它们相对于帧坐标的位置，以及从一帧到另一帧跟踪物体，提供令人印象深刻的精度。然而，为了提供可靠的结果，目前的目标检测技术远非实时的，并且需要大量的硬件资源。目前的技术使我们能够在帧级应用视频分类/检测和跟踪算法从视频中提取模式。这为数据管理提供了机会。我们的研究利用从视频中提取的模式探索流视频源的声明性查询处理。深度学习原语（例如，对象检测）可以在声明性框架中作为用户定义函数（UDF）实现。我们的研究探索了视频流上与查询相关的对象的有效识别，利用按需构建的查询特定网络（过滤器），而不是大型和更昂贵的深度学习模型。使用更便宜的查询特定模型处理帧，可以快速删除与查询不相关的帧，并可以显着提高帧处理速率。我们将探索应用此类过滤器的选项，并探索在速度和准确性之间的优化权衡。对检测对象的时空查询处理具有重要的意义。在监控应用程序中，查询可以使用空间（例如，人在汽车前面等）和/或时间（例如，汽车旁边到停车标志10分钟）约束来表示。同时，物体之间的相互作用（例如，人类打碎玻璃）是普遍存在的。本研究研究了当涉及帧中检测到的对象时，如何在流视频场景中表达和评估这些谓词（空间、时间、动作、交互）。我们计划在涉及视频对象的谓词上支持一般合取范式查询。最后但并非最不重要的是，我们扩展了框架以支持多个视频流，而不是单个视频流。这为在空间和时间约束下跨流关联和匹配对象以及相关优化提出了许多有趣的研究方向。从培训的角度来看，这项研究是及时的。它融合了数据管理和深度学习原语，构建了一个实时视频查询处理系统。