Multimodal Video Search by Examples (MVSE)

如何有效和高效地从大型视频档案（如BBC电视节目）中搜索内容是一个重大挑战。搜索通常是通过使用预定义的元数据，如标题，标签和观众的笔记关键字查询。然而，很难使用关键字来搜索视频中特定发言者在特定位置谈论特定主题的特定时刻。大多数视频几乎没有关于视频内容的元数据，并且自动元数据提取还不够可靠。此外，元数据可能会随时间而变化，无法涵盖所有内容。因此，按关键字搜索对于全面且持久的视频搜索解决方案来说不是理想的方法。通过示例的视频搜索是期望的替代方案，因为它允许通过感兴趣内容的一个或多个示例来搜索内容，而不必指定对关键字的兴趣。然而，视频搜索的例子是众所周知的挑战，其性能仍然很差。为了提高搜索性能，应该考虑多种形式-图像，声音，语音和文本，因为每种形式都提供了单独的搜索线索，因此多个线索应该识别更相关的内容。多模式视频搜索（MVSE）这是一个新兴的研究领域，目前的技术水平还远远不够理想，所以还有很长的路要走。MVSE没有商业服务。该提案是通过BBC数据科学合作伙伴关系与BBC研发部门共同创建的，通过多次在线会议和一次涉及所有合作伙伴的面对面会议。该提案是根据最近未发表的关于BBC现任工作人员（制片人、记者、档案管理员）如何搜索媒体内容的人种学研究提出的。结果发现，他们非常感兴趣的知识检索档案或其他来源，但他们需要更丰富的元数据和编目的非语言数据。在这项提案中，我们将研究高效，有效，可扩展和强大的MVSE，其中视频档案是大型的，历史的和动态的;和形式的人（脸或声音），上下文和主题。目的是为MVSE开发一个框架，并通过开发一个原型搜索工具对其进行验证。这样的搜索工具将对BBC和大英图书馆等组织有用，他们拥有大量的视频档案收藏，并希望为自己的员工和公众提供搜索工具。对于像Youtube这样的公司来说，它也是有用的，他们托管来自公众的视频，并希望通过例子来实现视频搜索。我们将解决开发高效，有效，可扩展和强大的MVSE解决方案的关键挑战，包括视频分割，内容表示，哈希，排名和融合。该提案计划为期三年，涉及三个机构（剑桥，萨里，阿尔斯特）和一个合作伙伴（英国广播公司），他们将为该项目提供大量资源（估计为128.4万英镑）（见英国广播公司的支持信）。

How to effectively and efficiently search for content from large video archives such as BBC TV programmes is a significant challenge. Search is typically done via keyword queries using pre-defined metadata such as titles, tags and viewer's notes. However, it is difficult to use keywords to search for specific moments in a video where a particular speaker talks about a specific topic at a particular location. Most videos have little or no metadata about content in the video, and automatic metadata extraction is not yet sufficiently reliable. Furthermore, metadata may change over time and cannot cover all content. Therefore, search by keyword is not a desirable approach for a comprehensive and long-lasting video search solution. Video search by examples is a desirable alternative as it allows search for content by one or more examples of the interested content without having to specify interest in keyword. However, video search by examples is notoriously challenging, and its performance is still poor. To improve search performance, multiple modalities should be considered - image, sound, voice and text, as each modality provides a separate search cue so multiple cues should identify more relevant content. This is multimodal video search by examples (MVSE). This is an emerging area of research, and the current state of the art is far from desirable so there is a long way to go. There is no commercial service for MVSE.This proposal has been co-created with BBC R&D through the BBC Data Science Partnership via a number of online meetings and one face to face meeting involving all partners. The proposal has been informed by recent unpublished ethnographic research on how current BBC staff (producers, journalists, archivists) search for media content. It was found that they were very interested in knowledge retrieval from archives or other sources but they required richer metadata and cataloguing of non-verbal data. In this proposal we will study efficient, effective, scalable and robust MVSE where video archives are large, historical and dynamic; and the modalities are person (face or voice), context, and topic. The aim is to develop a framework for MVSE and validate it through the development of a prototype search tool. Such a search tool will be useful for organisations such as the BBC and British Library, who maintain large collections of video archives and want to provide a search tool for their own staff as well as for the public. It will also be useful for companies such as Youtube who host videos from the public and want to enable video search by examples. We will address key challenges in the development of an efficient, effective, scalable and robust MVSE solution, including video segmentation, content representation, hashing, ranking and fusion. This proposal is planned for three years, involving three institutions (Cambridge, Surrey, Ulster) and one partner (the BBC) who will contribute significant resources (estimated at £128.4k) to the project (see Letter of Support from the BBC).