Systemization of audio-visual knowledge resources using graphical models

最近计算机技术的进步，特别是存储技术的进步，使视听知识资源的数量和质量都有了显著提高。这些资源中的大多数没有配备索引信息，因此，普通人很难浏览每个数据库的全部内容。人们对视听知识资源的系统化和利用技术提出了强烈的要求。然而，统计模式识别技术还没有达到足够的性能，这一目的。此外，并不总是清楚什么类型的索引是有用的。在这项研究中，我们采取了一种方法，这些数据库索引以不同的方式与无监督的方式，并提取这些标签之间的依赖关系。首先，我们对棒球视频进行了场景识别。我们与NHK科学技术研究实验室一起构建了43场美国职业棒球大联盟比赛的注释数据库，并将其用于我们的评估。我们用了瓦里 ...更多信息 场景标签之间的关系，如场景上下文，以及统一的音频和视频信息。我们取得了60%的准确率为16个场景识别和90%的召回率分数场景检测。我们的技术有望为广播公司的自动高光提取系统做出贡献。其次，我们参加了美国NIST组织的TRECVID研讨会，研究了高层特征提取任务。我们通过对视频特征的无监督聚类构建了“视觉词”的树结构字典，并为每个词选择了一个树切割作为字典。通过使用词袋方法，我们构建了一个强大的提取系统对数据量的差异，为每个功能。我们还提取了有效的“运动词”的动态特征。我们的方法在提取39个特征的任务中取得了显着的改进。其他研究课题包括基于图形模型的鲁棒语音识别、异步多模态输入的多模态接口、人体步态建模等。少

Recent advances in computer technology, particularly in storage technology, have resulted in significant increases in the number and quality of audio-visual knowledge resources. Most of those resources are not equipped with index information, and thus, it has become difficult for ordinary people to browse the entire content of each database. Techniques for systemizing audio-visual knowledge resources and utilizing them have been strongly demanded. However, statistical pattern recognition techniques have not yet achieved enough performance for this purpose. In addition, it is not always clear what kinds of indexing are useful. In this study, we take an approach to index those databases in different ways with unsupervised manner, and extract dependencies among those labels. First, we carried scene recognition for baseball video. We constructed annotated database for 43 games of Major League Baseball with NHK Science & Technical Research Labs and used them for our evaluation. We used vari … More ous relationships between scene labels such as scene contexts, and unified audio and visual information. We achieved 60% accuracy for 16 scene recognition and 90% recall rate for score scene detection. Our techniques are expected to contribute much to make automatic highlight extraction systems for broadcast companies. Second, we participated in TRECVID workshop organized by NIST, USA, to study the high-level feature extraction task. We constructed tree-structured dictionaries of "visual words" by unsupervised clustering for video features, and selected a tree-cut as a dictionary for each word. By using Bag-of-word approach, we constructed a robust extraction system against the differences in data amount for each feature. We also extracted effective "motion words" for dynamic features. Our method achieved significant improvements in the task of extracting 39 features. The other research topics include robust speech recognition using graphical models, multi-modal interface for asynchronous multi-modal inputs, human-gait modeling. Less