Reverse Storyboarding for Video Content Based Retrieval

该提案将探索使用情节提要草图作为自动索引和交互查询大型视频数据库的一种手段。为此，我们借鉴了动画和计算机图形学的技术，解决了基于视频内容的图像检索(CBIR)的计算机视觉问题。故事板是一种强大的视觉脚本技术，动画师通常使用它来计划和交流其作品的内容。简单地说，故事板是一系列说明电影关键场景和事件的视觉图像。然而，故事板不仅仅是静态的草图-它们还使用从当代动画中借用的各种运动线索来描述动态；条纹和重影线、对象变形以及更传统的指示器，如箭头。就像艺术家的素描是对静态场景的空间抽象一样，故事板也可以被认为是描述图像序列(视频)中显著时刻的时空抽象。我们认为，故事板具有许多独特的属性，可以应用于视频CBIR。故事板提供了丰富的词汇表，可以用来简洁地提出搜索查询。它们可以表示一系列动力学，如振动、平移、旋转、碰撞和遮挡。以前时间约束在CBIR中的使用主要限于对象轨迹和场景过渡，我们相信指定额外的动态约束的能力将有助于提高搜索精度和性能。故事板也是一种高度传统和可理解的叙事形式-例如，漫画书被用来向所有年龄段的非技术性观众传达故事。因此，我们相信故事板还将为视频CBIR提供一个自然和可用的界面。拟议的研究将基于PI最近在视频样式方面的工作；特别是将视频渲染为卡通。我们将建立在我们的Video Paintbox软件的基础上，使我们能够从真实视频片段中推断出的运动提示的色域多样化。这些技术将被用来从存储在视频数据库中的剪辑中解析动态提示。我们将开发一个用于视频中动态线索表示的本体，并开发基于空间和动态内容的搜索查询与数据库匹配的新算法。这项工作将被整合，以产生一个可演示的故事板驱动的CBIR系统。该系统将使我们能够测试我们的假设，即情节提要草图的直观可理解的性质，结合其紧凑的表示和丰富的动态词汇，可以被用来提高视频CBIR的可用性、准确性和性能。

This proposal will explore the use of storyboard sketches as a means of automatically indexing and interactively querying large video databases. In doing so we address the Computer Vision problem of video Content Based Image Retrieval (CBIR) using techniques borrowed from animation and Computer Graphics.Storyboards are a powerful visual scripting technique commonly used by animators to plan and communicate the content of their productions. In simple terms, a storyboard is a series of visual images that illustrate a film's key scenes and events. However storyboards are more than static sketches --- they also depict dynamics using a variety of motion cues borrowed from contemporary animation; streak and ghosting lines, object deformations as well as more conventional indicators such as arrow-heads. Much as an artist's sketch is spatial abstraction of a static scene, so a storyboard may be considered a spatio-temporal abstraction depicting salient instants within a image sequence (video).We argue that storyboards have a number of unique properties that can be brought to bear on video CBIR. Storyboards confer a rich vocabulary with which to concisely pose search queries. They can represent a range of dynamics such as oscillation, translation, rotation, collision and occlusion. Previous use of temporal constraints in CBIR has been limited mainly to object trajectories and scene transitions, and we believe that the ability to specify additional dynamic constraints will help to improve search accuracy and performance. Storyboards are also a highly conventional and comprehensible narrative form --- comic books, for example, are used to convey stories to non-technical audiences of all ages. Thus we believe that storyboards will also provide a natural and usable interface for video CBIR.The proposed research will build upon the PI's recent work on video stylisation; specifically on rendering video into cartoons. We will build upon our Video Paintbox software to diversify the gamut of motion cues we are able to infer from real video footage. These techniques will be harnessed to parse dynamic cues from clips stored in video databases. We will develop an ontology for the representation of dynamic cues in video, and develop novel algorithms for matching search queries to the database based on spatial and dynamic content. This work will be integrated to produce a demonstrable storyboard driven CBIR system. This system will enable us test our hypothesis that the intuitively comprehensible nature of storyboard sketches, combined with their compact representation and rich vocabulary of dynamics, may be leveraged to enhance usability, accuracy and performance of video CBIR.