III: Small: Query-By-Sketch: Simplifying Video Clip Retrieval Through A Visual Query Paradigm

III：小：按草图查询：通过可视化查询范式简化视频剪辑检索

• 批准号: 2335881
• 负责人: Kexin Rong
• 金额: $ 60万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-04-15 至 2027-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2335881&HistoricalAwards=false
• 关键词: III; Small; Query; Sketch; Simplifying

This project addresses the growing demand for analyzing movement patterns in videos across diverse applications such as sports analytics, wildlife tracking, urban planning, and autonomous vehicle development. For example, analyzing vehicle trajectories from surveillance videos is essential for improving traffic safety. This project introduces a novel method for querying movement patterns in videos, enabling users to sketch events of interest on a canvas. The main innovation lies in accurately and efficiently matching free-form sketches to real-world trajectories, overcoming challenges posed by ambiguous user intent and variations in perspective, orientation, and camera movements. Consider a user describing a left-turning vehicle event as a 90-degree angle from a top-down perspective; in practice, the turning angles may appear different on video due to varying camera positions relative to vehicles. The project will lead to an open-source video database featuring a sketch-based query interface, making the analysis of movement patterns in videos more accessible and accurate. Research findings will be disseminated through publications at top conferences and incorporated into new database courses at Georgia Tech, as well as research classes for Atlanta-area high school girls interested in pursuing computing careers. Video retrieval from trajectory queries has been explored by the database and machine learning communities using SQL-like and natural language interfaces, but they face limitations due to high query specification time or poor generalizability to unseen videos. This project seeks to address these challenges by introducing a novel visual query paradigm that enables users to sketch exploratory trajectory queries in video analytics through drag-and-drop actions. The project is structured around two research thrusts. The first focuses on developing a human-in-loop similarity search framework that leverages active-learning techniques to solicit user feedback. This process aims to clarify user intent in query specifications and address inaccuracies inherent in human sketching. Domain-specific knowledge will be incorporated as additional predicates in the pre-processing and post-processing stages of similarity search to further enhance retrieval efficiency and quality. The second thrust develops an end-to-end machine learning model that learns a robust similarity measure between user-drawn sketches and trajectories in real-world videos, accounting for variations in camera angles and movements. It will address the lack of diverse and labeled datasets for video retrieval from trajectory queries by developing a self-supervised learning framework based on trajectory simulation. Overall, this project will leverage database-style optimization to reduce both user effort and computational resources required for utilizing vision models in exploratory video analytics, which will help expand the adoption of video analytics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目满足了对视频中运动模式分析的日益增长的需求，这些运动模式涉及体育分析、野生动物跟踪、城市规划和自动驾驶汽车开发等各种应用。例如，从监控视频中分析车辆轨迹对于提高交通安全至关重要。该项目介绍了一种新颖的方法，用于查询视频中的运动模式，使用户能够在画布上绘制感兴趣的事件。主要创新在于准确、高效地将自由形式的草图与真实世界的轨迹进行匹配，克服用户意图模糊以及视角、方向和相机移动变化带来的挑战。考虑用户从自上而下的视角将左转车辆事件描述为90度角;在实践中，由于相对于车辆的不同相机位置，转弯角度可能在视频上看起来不同。该项目将建立一个开源视频数据库，该数据库具有基于草图的查询界面，使视频中的运动模式分析更容易获得和准确。研究结果将通过在顶级会议上的出版物传播，并纳入格鲁吉亚理工学院的新数据库课程，以及为有兴趣从事计算机职业的亚特兰大地区高中女生开设的研究课程。数据库和机器学习社区已经使用类似SQL和自然语言接口探索了从轨迹查询中检索视频，但由于查询规范时间长或对未见过视频的泛化能力差，它们面临限制。该项目旨在通过引入一种新颖的可视化查询范式来解决这些挑战，该范式使用户能够通过拖放操作在视频分析中绘制探索性轨迹查询。该项目围绕两个研究重点展开。第一个重点是开发一个人在回路相似性搜索框架，利用主动学习技术征求用户反馈。此过程旨在澄清用户在查询规范中的意图，并解决人类草图中固有的不准确性。在相似性检索的前处理和后处理阶段，特定领域的知识将作为额外的谓词被纳入，以进一步提高检索效率和质量。第二个重点是开发一个端到端的机器学习模型，该模型可以学习用户绘制的草图和真实视频中的轨迹之间的鲁棒相似性度量，并考虑到摄像机角度和运动的变化。它将通过开发一个基于轨迹模拟的自监督学习框架来解决缺乏多样性和标记数据集的问题。总体而言，该项目将利用数据库式优化，减少用户在探索性视频分析中使用视觉模型所需的工作量和计算资源，这将有助于扩大视频分析的采用。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。