Multimodal Video Search by Examples (MVSE)

如何有效、高效地从BBC电视节目等大型视频档案中搜索内容是一个重大挑战。搜索通常通过使用预定义元数据（如标题、标签和查看者的注释）的关键字查询完成。然而，很难使用关键词来搜索视频中的特定时刻，即特定演讲者在特定地点谈论特定主题。大多数视频中很少或没有关于视频内容的元数据，自动元数据提取还不够可靠。此外，元数据可能会随时间而变化，无法覆盖所有内容。因此，对于一个全面和持久的视频搜索解决方案来说，通过关键字搜索并不是一个理想的方法。通过示例搜索视频是一种理想的选择，因为它允许通过感兴趣的内容的一个或多个示例来搜索内容，而不必指定关键字中的兴趣。然而，基于示例的视频搜索是出了名的具有挑战性，而且它的性能仍然很差。为了提高搜索性能，应该考虑多种模式——图像、声音、语音和文本，因为每种模式都提供了一个单独的搜索线索，因此多个线索应该识别更多相关的内容。这就是多模态示例视频搜索（MVSE）。这是一个新兴的研究领域，目前的技术水平还远远不够理想，所以还有很长的路要走。MVSE没有商业服务。该提案是与BBC研发部门通过BBC数据科学合作伙伴关系通过多次在线会议和一次涉及所有合作伙伴的面对面会议共同制定的。这项提议是由最近未发表的一项关于BBC员工（制片人、记者、档案保管员）如何搜索媒体内容的人种学研究得出的。研究发现，他们对从档案或其他来源检索知识非常感兴趣，但他们需要更丰富的元数据和非语言数据的编目。在本提案中，我们将研究高效、有效、可扩展和健壮的MVSE，其中视频档案是大的、历史的和动态的；模态包括人（脸或声音）、语境和话题。目的是为MVSE开发一个框架，并通过开发一个原型搜索工具来验证它。这样的搜索工具对BBC和大英图书馆这样的组织很有用，它们保存着大量的视频档案，希望为自己的员工和公众提供一个搜索工具。它对Youtube等公司也很有用，这些公司托管来自公众的视频，并希望通过示例实现视频搜索。我们将解决开发高效、有效、可扩展和健壮的MVSE解决方案的关键挑战，包括视频分割、内容表示、哈希、排名和融合。该提案计划为期三年，涉及三个机构（剑桥，萨里，阿尔斯特）和一个合作伙伴（BBC），他们将为该项目提供大量资源（估计为128.4万英镑）（见BBC的支持信）。

How to effectively and efficiently search for content from large video archives such as BBC TV programmes is a significant challenge. Search is typically done via keyword queries using pre-defined metadata such as titles, tags and viewer's notes. However, it is difficult to use keywords to search for specific moments in a video where a particular speaker talks about a specific topic at a particular location. Most videos have little or no metadata about content in the video, and automatic metadata extraction is not yet sufficiently reliable. Furthermore, metadata may change over time and cannot cover all content. Therefore, search by keyword is not a desirable approach for a comprehensive and long-lasting video search solution. Video search by examples is a desirable alternative as it allows search for content by one or more examples of the interested content without having to specify interest in keyword. However, video search by examples is notoriously challenging, and its performance is still poor. To improve search performance, multiple modalities should be considered - image, sound, voice and text, as each modality provides a separate search cue so multiple cues should identify more relevant content. This is multimodal video search by examples (MVSE). This is an emerging area of research, and the current state of the art is far from desirable so there is a long way to go. There is no commercial service for MVSE.This proposal has been co-created with BBC R&D through the BBC Data Science Partnership via a number of online meetings and one face to face meeting involving all partners. The proposal has been informed by recent unpublished ethnographic research on how current BBC staff (producers, journalists, archivists) search for media content. It was found that they were very interested in knowledge retrieval from archives or other sources but they required richer metadata and cataloguing of non-verbal data. In this proposal we will study efficient, effective, scalable and robust MVSE where video archives are large, historical and dynamic; and the modalities are person (face or voice), context, and topic. The aim is to develop a framework for MVSE and validate it through the development of a prototype search tool. Such a search tool will be useful for organisations such as the BBC and British Library, who maintain large collections of video archives and want to provide a search tool for their own staff as well as for the public. It will also be useful for companies such as Youtube who host videos from the public and want to enable video search by examples. We will address key challenges in the development of an efficient, effective, scalable and robust MVSE solution, including video segmentation, content representation, hashing, ranking and fusion. This proposal is planned for three years, involving three institutions (Cambridge, Surrey, Ulster) and one partner (the BBC) who will contribute significant resources (estimated at £128.4k) to the project (see Letter of Support from the BBC).