Network Based Organization of Continuous Media by Intermodal Collaboration and Domain Knowledge Use

本研究项目的目标是：(1)通过多通道协作(即对每个视频流进行协同分析)，开发由视觉、听觉、文本和图形流组成的视频流等连续媒体的内容分析方法；(2)建立便于内容分析的领域知识库；(3)设计一种基于网络的媒体组织方案，称为事件网。本研究的主要成果分为三个问题。基于多通道协作的视频索引和标注为了自动生成事件网的节点，我们体现了几种基于多通道协作的内容分析方法。特别是，通过考虑多模流之间的对应关系，开发了针对广播体育视频的基于事件的索引和基于运动员和动作的语义标注。我们验证了多式联运协作在效率和可靠性方面优于单流分析。基于事件之间关系的媒体组织我们考虑了分析与事件网络的节点相对应的事件关系的方法。具体地说，我们提出了一种将体育视频中的直播和回放场景联系起来的方法，重点关注了相同和相似的关系。媒体组织的应用：视频摘要和视频门户通过从事件网中提取精彩事件，提出了基于精彩的视频摘要方法。实验结果表明，一小时长的视频可以压缩成一分钟长的个性化摘要。此外，我们还构建了视频门户，作为媒体组织的一个应用来支持我们访问多媒体空间的工具。

The goal of this research project is as follows: (1) to develop contents analysis method of continuous media such as video streams consisting of visual, auditory, textual and graphical streams, by means of intermodal collaboration that means collaborative analysis of each video stream, (2) to make the domain knowledge base to facilitate the contents analysis, and (3) to design a scheme of network based media organization called an event net. The major results of this research are classified into three issues.1. Video indexing and annotation by intermodal collaborationTo automatically produce nodes of the event net, we embodied a couple of contents analysis methods by intermodal collaboration. In particular, event based indexing for broadcasted sports video and semantical annotation in terms of players and actions were developed by considering the correspondence between multimodal streams. We verified that intermodal collaboration outperforms single stream analysis with respect to efficiency and reliability.2. Media organization based on relations between eventsWe considered approaches to analysis of relations of events that correspond to nodes of the event net. Specifically, we proposed a method of linking live and replay scenes in the broadcasted sports video, focusing on identical and similar relations.3. Application of media organization: video abstraction and video portalBy extracting highlight events from the event net, highlight based video abstraction method was pro- posed. The experimental results demonstrated that an hour-length video can be compressed into a minute-length personalized abstract. In addition, we constructed video portal that is a tool to support our access to a multimedia space as an application of media organization.