Reverse Storyboarding for Video Content Based Retrieval

用于基于视频内容的检索的反向故事板

• 批准号: EP/D055032/2
• 负责人: John Collomosse
• 金额: --
• 依托单位: University of Surrey
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2009
• 资助国家: 英国
• 起止时间: 2009 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FD055032%2F2
• 关键词: Reverse; Storyboarding; Video; Content; Based

This proposal will explore the use of storyboard sketches as a means of automatically indexing and interactively querying large video databases. In doing so we address the Computer Vision problem of video Content Based Image Retrieval (CBIR) using techniques borrowed from animation and Computer Graphics.Storyboards are a powerful visual scripting technique commonly used by animators to plan and communicate the content of their productions. In simple terms, a storyboard is a series of visual images that illustrate a film's key scenes and events. However storyboards are more than static sketches --- they also depict dynamics using a variety of motion cues borrowed from contemporary animation; streak and ghosting lines, object deformations as well as more conventional indicators such as arrow-heads. Much as an artist's sketch is spatial abstraction of a static scene, so a storyboard may be considered a spatio-temporal abstraction depicting salient instants within a image sequence (video).We argue that storyboards have a number of unique properties that can be brought to bear on video CBIR. Storyboards confer a rich vocabulary with which to concisely pose search queries. They can represent a range of dynamics such as oscillation, translation, rotation, collision and occlusion. Previous use of temporal constraints in CBIR has been limited mainly to object trajectories and scene transitions, and we believe that the ability to specify additional dynamic constraints will help to improve search accuracy and performance. Storyboards are also a highly conventional and comprehensible narrative form --- comic books, for example, are used to convey stories to non-technical audiences of all ages. Thus we believe that storyboards will also provide a natural and usable interface for video CBIR.The proposed research will build upon the PI's recent work on video stylisation; specifically on rendering video into cartoons. We will build upon our Video Paintbox software to diversify the gamut of motion cues we are able to infer from real video footage. These techniques will be harnessed to parse dynamic cues from clips stored in video databases. We will develop an ontology for the representation of dynamic cues in video, and develop novel algorithms for matching search queries to the database based on spatial and dynamic content. This work will be integrated to produce a demonstrable storyboard driven CBIR system. This system will enable us test our hypothesis that the intuitively comprehensible nature of storyboard sketches, combined with their compact representation and rich vocabulary of dynamics, may be leveraged to enhance usability, accuracy and performance of video CBIR.

本提案将探讨使用故事板草图作为自动索引和交互式查询大型视频数据库的一种手段。在这样做的过程中，我们解决了计算机视觉问题的视频内容为基础的图像检索（CBIR）使用的技术借鉴动画和计算机Graphics.Storyboards是一个强大的可视化脚本技术，通常用于动画师计划和沟通的内容，他们的产品。简单地说，故事板是一系列的视觉图像，说明了电影的关键场景和事件。然而，故事板不仅仅是静态的草图-它们还使用从当代动画中借来的各种运动线索来描绘动态;条纹和重影线，对象变形以及更传统的指示器，如箭头。正如艺术家的素描是一个静态场景的空间抽象，所以故事板可以被认为是一个时空的抽象，描绘图像序列（视频）内的突出瞬间。我们认为，故事板有一些独特的属性，可以带来承担视频CBIR。情节串联图板提供了丰富的词汇，可以用来简洁地提出搜索查询。它们可以代表一系列的动力学，如振荡，平移，旋转，碰撞和闭塞。以前在CBIR中使用的时间约束主要限于对象轨迹和场景转换，我们相信，指定额外的动态约束的能力将有助于提高搜索的准确性和性能。故事情节串联图板也是一种高度传统和易于理解的叙事形式-例如，漫画书被用来向所有年龄段的非技术观众传达故事。因此，我们相信，故事板也将提供一个自然和可用的接口视频CBIR。拟议的研究将建立在PI的最近的工作，视频风格化，特别是对渲染视频成卡通。我们将建立在我们的视频Paintbox软件，以多样化的运动线索，我们能够推断从真实的视频画面的色域。这些技术将被利用来解析存储在视频数据库中的剪辑的动态线索。我们将开发一个本体的视频中的动态线索的表示，并开发新的算法匹配搜索查询的数据库的基础上的空间和动态内容。这项工作将被整合，以产生一个可演示的故事板驱动的CBIR系统。该系统将使我们能够测试我们的假设，即故事板草图的直观可理解的性质，结合其紧凑的表示和丰富的动态词汇，可以利用，以提高视频CBIR的可用性，准确性和性能。

项目成果

A bag-of-regions approach to sketch-based image retrieval

• DOI: 10.1109/icip.2011.6116513
• 发表时间: 2011-12
• 期刊: 2011 18th IEEE International Conference on Image Processing
• 作者: Rui Hu;T. Wang;J. Collomosse

Storyboard sketches for Content Based Video Retrieval

• DOI: 10.1109/iccv.2009.5459258
• 发表时间: 2009-09
• 期刊: 2009 IEEE 12th International Conference on Computer Vision
• 作者: J. Collomosse;Graham Mcneill;Yu Qian

Gradient field descriptor for sketch based retrieval and localization

用于基于草图的检索和定位的梯度场描述符

• DOI: 10.1109/icip.2010.5649331
• 发表时间: 2010
• 作者: Hu R

Motion-sketch Based Video Retrieval Using a Trellis Levenshtein Distance

使用网格编辑距离的基于运动草图的视频检索

• DOI: 10.1109/icpr.2010.38
• 发表时间: 2010
• 作者: Hu R

A performance evaluation of gradient field HOG descriptor for sketch based image retrieval

• DOI: 10.1016/j.cviu.2013.02.005
• 发表时间: 2013-07-01
• 期刊: COMPUTER VISION AND IMAGE UNDERSTANDING
• 影响因子: 4.5
• 作者: Hu, Rui;Collomosse, John
• 通讯作者: Collomosse, John