SBIR Phase I: Multimodal Semantic Video Retrieval and Summarization

这项小企业创新研究（SBIR）第一阶段项目旨在开发用于高度准确和高效的语义视频检索和总结的方法和软件。视频语义将提供满足用户偏好的个性化视频内容摘要。这些摘要将基于镜头粒度，而不是广泛使用的基于关键帧的、忽略语义的摘要。此外，所提出的技术将大大增强在线视频搜索，使用户能够只检索语义相关的镜头，而不是整个视频。该软件的关键组件是一个自动化的语义视频注释系统，该系统集成了实时视频镜头检测、语音识别、自然语言处理和逻辑推理等方法，可以根据用户的语义请求和偏好准确选择视频镜头。消费者和视频内容服务提供商将使用提出的自适应视频消息技术，使用语义视频摘要消息（SVS）有效地交流查询、偏好和结果。这个被提议的软件一旦商业化，将会改变在线视频内容的搜索和检索方式。此外，如果成功，该软件将推进构建视频摘要的最先进技术，与当前技术相比，它可以准确地响应语义级用户的查询。因此，该软件可能会引起许多内容提供商和消费者的兴趣，以便在众多视频应用程序中使用。该软件还可以集成到一直流行的数字视频录像机中，使所有者能够搜索大量存档的视频，并根据语义查询检索特定的视频，而不是通常不准确的文件名。另一方面，内容/服务提供商可以使用软件的独特摘要功能，其中用户可以预定义个性化的、基于语义的摘要标准，以便内容提供商自适应地（基于网络和设备功能）将匹配用户需求的摘要流式传输到他们的智能手机或其他移动设备。该奖项是根据2009年美国复苏和再投资法案（公法111-5）资助的。

This Small Business Innovation Research (SBIR) Phase I project aims to develop methodology and software for highly accurate and efficient semantic video retrieval and summarization. Video Semantics will, provide personalized summaries of video content that meet users' preferences. These summaries will be based on shot granularity instead of the widely used key-frame-based summaries that are oblivious of semantics. Additionally, the proposed technology will significantly enhance online video search by enabling users to retrieve only semantically-relevant shots instead of the entire video. The key component of the software is an automated semantic video annotation system that integrates realtime video shot detection, speech recognition, natural language processing, and logic inference methods to accurately select video shots according to semantic user requests and preferences. Consumers and video content service providers will use the proposed adaptive video messaging technique to efficiently communicate queries, preferences and results using Semantic Video Summary messages (SVS). The proposed software, once commercialized, can affect a shift in the way online video content is searched and retrieved. Moreover, if successful, the software will advance the state-of-the-art of constructing video summaries, which in contrast to current technologies, accurately responds to semantic level user queries. Consequently, the software may be of interest to numerous content providers and consumers to be employed in a multitude of video applications. The software could also be integrated into the ever-popular digital video recorders to enable the owner to search large volumes of archived videos and retrieve specific ones given semantic queries, rather than the usually inaccurate file names. On the other hand, the unique summarization capabilities of the software can be used by content/service providers where personalized, semantic-based summary criteria can be predefined by the user so that the content providers, adaptively (based on network and device capabilities) stream summaries matching users' requirements to their smart phones of other mobile devices.This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).