RI: Small: Collaborative Research: Ontology based Perceptual Organization of Audio-Video Events using Pattern Theory

RI：小型：协作研究：使用模式理论对音频-视频事件进行基于本体的感知组织

• 批准号: 1217676
• 负责人: Sudeep Sarkar
• 金额: $ 24.98万
• 依托单位: University of South Florida
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1217676&HistoricalAwards=false
• 关键词: RI; Small; Collaborative; Research; Ontology

It is natural that events of interest in observed scenes manifest themselves across multiple sensing modalities - vision, hearing, smell, etc. The remarkable perceptions of audio-video signals by natural systems, such as humans, also points to superiority of inferences drawn across modalities. It, therefore, seems natural to enhance performance of automated systems by using joint, cross-modal statistical inferences. However, the detection, organization, and understanding of cues and events in real-world scenarios are difficult tasks. This project seeks to develop a pattern-theoretic framework for achieving these goals. The main research items are: (1) development of mathematical quantities to represent audio and visual events and their spatiotemporal relations, (2) use domain-specific ontologies to impose semantic structure and to incorporate prior knowledge, and (3) derive algorithms for Bayesian inferences using efficient adaptations of Markov Chain Monte Carlo sampling. The use of pattern theory allows bridging of gaps between raw signals and high-level, domain-dependent semantics, and helps discovers large groups of audio-visual events likely to represent the same underlying event. This effort combines ideas from perceptual organization in computer vision, computational analysis of auditory signals, pattern theory, and prior developments in ontological structures. The methods developed here are applicable to many scenarios that deploy audio and video sensors, including problems of audio annotations of videos, speaker tracking in teleconferencing, and separation of multiple objects in remote surveillance. Broader impact activities involve the development of teaching modules, innovation and entrepreneurial training of the students, and communication of the findings to the community.

在观察到的场景中，感兴趣的事件自然会通过多种感知方式表现出来——视觉、听觉、嗅觉等。自然系统（如人类）对音频-视频信号的非凡感知也表明了跨模态推断的优越性。因此，通过使用联合、跨模态统计推断来提高自动化系统的性能似乎是很自然的。然而，在现实场景中对线索和事件的检测、组织和理解是一项困难的任务。本项目旨在开发一种模式理论框架来实现这些目标。主要研究项目有：(1)发展数学量来表示音频和视觉事件及其时空关系；(2)使用特定领域的本体来施加语义结构并整合先验知识；(3)利用马尔可夫链蒙特卡罗采样的有效适应推导贝叶斯推理算法。模式理论的使用可以弥合原始信号和高级领域相关语义之间的差距，并有助于发现可能表示相同底层事件的大量视听事件组。这项工作结合了计算机视觉中的感知组织、听觉信号的计算分析、模式理论和本体论结构的先前发展。本文开发的方法适用于部署音频和视频传感器的许多场景，包括视频的音频注释问题、电话会议中的说话人跟踪问题、远程监控中的多目标分离问题。更广泛的影响活动包括教学模块的发展，学生的创新和创业培训，以及向社区传播研究结果。