RI: Small: Collaborative Research: Ontology based Perceptual Organization of Audio-Video Events using Pattern Theory

这是很自然的，在观察到的场景中的感兴趣的事件表现自己在多个传感模态-视觉，听觉，嗅觉等显着的感知自然系统，如人类的音视频信号，也指出了优越性的推断得出跨模态。因此，通过使用联合的、跨模态的统计推断来增强自动化系统的性能似乎是很自然的。然而，在现实世界的场景中的线索和事件的检测，组织和理解是困难的任务。这个项目旨在开发一个模式理论框架来实现这些目标。主要研究项目有：（1）开发数学量来表示音频和视频事件及其时空关系，（2）使用特定领域的本体来施加语义结构并结合先验知识，以及（3）使用马尔可夫链蒙特卡罗采样的有效适应来导出贝叶斯推断的算法。模式理论的使用可以弥合原始信号和高层次的，域相关的语义之间的差距，并有助于发现大组的视听事件可能代表相同的底层事件。这项工作结合了计算机视觉中的感知组织，听觉信号的计算分析，模式理论和本体结构的先前发展的想法。这里开发的方法适用于部署音频和视频传感器的许多场景，包括视频的音频注释，电话会议中的扬声器跟踪，以及远程监控中的多个对象的分离等问题。影响更广泛的活动包括开发教学模块、对学生进行创新和创业培训以及向社区传播研究结果。

It is natural that events of interest in observed scenes manifest themselves across multiple sensing modalities - vision, hearing, smell, etc. The remarkable perceptions of audio-video signals by natural systems, such as humans, also points to superiority of inferences drawn across modalities. It, therefore, seems natural to enhance performance of automated systems by using joint, cross-modal statistical inferences. However, the detection, organization, and understanding of cues and events in real-world scenarios are difficult tasks. This project seeks to develop a pattern-theoretic framework for achieving these goals. The main research items are: (1) development of mathematical quantities to represent audio and visual events and their spatiotemporal relations, (2) use domain-specific ontologies to impose semantic structure and to incorporate prior knowledge, and (3) derive algorithms for Bayesian inferences using efficient adaptations of Markov Chain Monte Carlo sampling. The use of pattern theory allows bridging of gaps between raw signals and high-level, domain-dependent semantics, and helps discovers large groups of audio-visual events likely to represent the same underlying event. This effort combines ideas from perceptual organization in computer vision, computational analysis of auditory signals, pattern theory, and prior developments in ontological structures. The methods developed here are applicable to many scenarios that deploy audio and video sensors, including problems of audio annotations of videos, speaker tracking in teleconferencing, and separation of multiple objects in remote surveillance. Broader impact activities involve the development of teaching modules, innovation and entrepreneurial training of the students, and communication of the findings to the community.