Declarative Query Processing Over Real Time Video Streams

实时视频流上的声明式查询处理

• 批准号: RGPIN-2020-07238
• 负责人: Koudas, Nikolaos
• 金额: $ 2.11万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=741676
• 关键词: Declarative; Query; Processing; Over; Real

In the last few years, Deep Learning (DL) has become a dominant artificial intelligence (AI) technology in industry and academia. It has managed to revolutionize certain important practical applications. Video data abound; as of this writing 500 hours of video are uploaded on Youtube every minute. Numerous applications benefit from advanced techniques to process and understand video content ranging from video surveillance and video monitoring applications, to news production and autonomous driving. State of the art DL algorithms assess the presence of specific objects in an image, assess their properties (e.g. color, texture), their location relative to the frame coordinates as well as track an object from frame to frame delivering impressive accuracy. In order to deliver solid results however, state of the art object detection techniques are far from real time and require significant hardware resources. Current technology, enables us to extract a schema from a video applying video classification/detection and tracking algorithms at the frame level. This presents an opportunity for data management. Our research explores declarative query processing on streaming video sources utilizing the schema extracted from video. Deep learning primitives (e.g., object detection) can be realized as User Defined Functions (UDF's) in a declarative framework. Our research explores the efficient identification of objects relevant to a query on the video streams, utilizing query specific networks (filters) that are built on demand, as opposed to large and more expensive deep learning models. Processing frames with cheaper query specific models, can quickly remove frames not relevant to the query and can dramatically improve frame processing rate. We explore options for applying such filters and explore optimization trade offs over speed and accuracy. Spatio-temporal query processing on detected objects is of vast importance. Queries can be expressed using spatial (e.g., human in-front-of car, etc) and/or temporal (e.g., car next-to stop-sign for 10 minutes) constraints in monitoring applications. At the same time interactions between objects (e.g., human breaking glass) are prevalent. This research studies how such predicates (spatial, temporal, actions, interactions) can be expressed and evaluated in a streaming video scenario when objects detected in frames are involved. We plan to support general conjunctive normal form queries on such predicates involving video objects. Last but not least, we extend the framework to support multiple video streams, as opposed to a single video stream. This raises numerous interesting research directions in correlating and matching objects across streams under spatial and temporal constraints and associated optimizations. This research is  timely from a training perspective. It merges data management and deep learning primitives to build a real time video query processing system.

在过去的几年里，深度学习（DL）已经成为工业界和学术界主导的人工智能（AI）技术。它已经成功地使某些重要的实际应用发生了革命性的变化。视频数据比比皆是;截至本文撰写时，每分钟有500小时的视频上传到Youtube上。从视频监控和视频监控应用到新闻制作和自动驾驶，许多应用都受益于先进的技术来处理和理解视频内容。最先进的DL算法评估图像中特定对象的存在，评估它们的属性（例如颜色，纹理），它们相对于帧坐标的位置，以及从帧到帧跟踪对象，提供令人印象深刻的准确性。然而，为了提供可靠的结果，现有技术的对象检测技术远非真实的时间，并且需要大量的硬件资源。目前的技术，使我们能够提取一个模式，从视频应用视频分类/检测和跟踪算法在帧级。这为数据管理提供了机会。我们的研究探讨了声明性查询处理流媒体视频源利用模式提取的视频。深度学习原语（例如，对象检测）可以被实现为声明性框架中的用户定义函数（UDF）。我们的研究探索了有效识别与视频流上的查询相关的对象，利用按需构建的查询特定网络（过滤器），而不是大型且更昂贵的深度学习模型。使用更便宜的查询特定模型处理帧，可以快速删除与查询无关的帧，并可以显著提高帧处理速率。我们探索应用这种过滤器的选项，并探索速度和准确性的优化权衡。检测对象的时空查询处理是非常重要的。可使用空间（例如，汽车前面的人等）和/或时间（例如，停车标志旁的汽车10分钟）的限制。同时，对象之间的交互（例如，人碎玻璃）是普遍的。本研究研究如何这样的谓词（空间，时间，动作，交互）可以表示和评估流视频场景中检测到的帧中的对象时，涉及。我们计划支持一般的合取范式查询这样的谓词涉及视频对象。最后但并非最不重要的是，我们扩展了框架以支持多个视频流，而不是单个视频流。这提出了许多有趣的研究方向，在空间和时间的约束和相关的优化下，跨流关联和匹配对象。 从培训的角度来看，这项研究是及时的。它融合了数据管理和深度学习原语，构建了一个真实的实时视频查询处理系统。