III: Small: Query-By-Sketch: Simplifying Video Clip Retrieval Through A Visual Query Paradigm

该项目满足了在体育分析、野生动物跟踪、城市规划和自动驾驶汽车开发等不同应用中分析视频运动模式的日益增长的需求。例如，从监控视频中分析车辆轨迹对于提高交通安全至关重要。这个项目引入了一种新的方法来查询视频中的运动模式，使用户能够在画布上绘制感兴趣的事件。主要的创新在于准确有效地将自由形式的草图与现实世界的轨迹相匹配，克服了模糊的用户意图和视角、方向和相机运动的变化所带来的挑战。假设用户从自上而下的角度描述一个90度角的左转车辆事件；实际上，由于摄像机相对于车辆的位置不同，在视频中可能出现不同的转弯角度。该项目将导致一个开源视频数据库，其特点是基于草图的查询界面，使视频中的运动模式分析更容易访问和准确。研究结果将通过顶级会议的出版物传播，并纳入佐治亚理工学院的新数据库课程，以及亚特兰大地区对从事计算机职业感兴趣的高中女生的研究课程。数据库和机器学习社区已经使用类sql和自然语言接口探索了轨迹查询中的视频检索，但由于查询规范时间长或对未见过的视频的泛化性差，它们面临限制。该项目旨在通过引入一种新颖的可视化查询范式来解决这些挑战，该范式使用户能够通过拖放操作在视频分析中绘制探索性轨迹查询。该项目围绕两个研究重点展开。第一个重点是开发一个人在循环相似性搜索框架，利用主动学习技术征求用户反馈。该过程旨在澄清查询规范中的用户意图，并解决人类素描固有的不准确性。在相似性搜索的预处理和后处理阶段，将特定领域知识作为附加谓词加入，进一步提高检索效率和质量。第二个重点是开发端到端机器学习模型，该模型学习用户绘制的草图和现实世界视频中的轨迹之间的强大相似性度量，考虑到摄像机角度和运动的变化。它将通过开发基于轨迹模拟的自监督学习框架来解决缺乏多样化和标记的数据集用于从轨迹查询中检索视频的问题。总的来说，该项目将利用数据库风格的优化来减少在探索性视频分析中使用视觉模型所需的用户努力和计算资源，这将有助于扩大视频分析的采用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project addresses the growing demand for analyzing movement patterns in videos across diverse applications such as sports analytics, wildlife tracking, urban planning, and autonomous vehicle development. For example, analyzing vehicle trajectories from surveillance videos is essential for improving traffic safety. This project introduces a novel method for querying movement patterns in videos, enabling users to sketch events of interest on a canvas. The main innovation lies in accurately and efficiently matching free-form sketches to real-world trajectories, overcoming challenges posed by ambiguous user intent and variations in perspective, orientation, and camera movements. Consider a user describing a left-turning vehicle event as a 90-degree angle from a top-down perspective; in practice, the turning angles may appear different on video due to varying camera positions relative to vehicles. The project will lead to an open-source video database featuring a sketch-based query interface, making the analysis of movement patterns in videos more accessible and accurate. Research findings will be disseminated through publications at top conferences and incorporated into new database courses at Georgia Tech, as well as research classes for Atlanta-area high school girls interested in pursuing computing careers. Video retrieval from trajectory queries has been explored by the database and machine learning communities using SQL-like and natural language interfaces, but they face limitations due to high query specification time or poor generalizability to unseen videos. This project seeks to address these challenges by introducing a novel visual query paradigm that enables users to sketch exploratory trajectory queries in video analytics through drag-and-drop actions. The project is structured around two research thrusts. The first focuses on developing a human-in-loop similarity search framework that leverages active-learning techniques to solicit user feedback. This process aims to clarify user intent in query specifications and address inaccuracies inherent in human sketching. Domain-specific knowledge will be incorporated as additional predicates in the pre-processing and post-processing stages of similarity search to further enhance retrieval efficiency and quality. The second thrust develops an end-to-end machine learning model that learns a robust similarity measure between user-drawn sketches and trajectories in real-world videos, accounting for variations in camera angles and movements. It will address the lack of diverse and labeled datasets for video retrieval from trajectory queries by developing a self-supervised learning framework based on trajectory simulation. Overall, this project will leverage database-style optimization to reduce both user effort and computational resources required for utilizing vision models in exploratory video analytics, which will help expand the adoption of video analytics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.