RI: Small: Collaborative Research: Ontology based Perceptual Organization of Audio-Video Events using Pattern Theory

在观察到的场景中感兴趣的事件自然会通过多种感知模式表现出来--视觉、听觉、嗅觉等。自然系统(如人类)对音频-视频信号的显著感知也表明了跨模式推断的优越性。因此，通过使用联合、跨模式统计推断来提高自动化系统的性能似乎是很自然的。然而，检测、组织和理解现实世界场景中的线索和事件是一项困难的任务。这个项目寻求开发一个模式理论框架来实现这些目标。主要的研究内容是：(1)发展表示视听事件及其时空关系的数学量；(2)使用特定领域的本体来施加语义结构并结合先验知识；(3)利用马尔可夫链蒙特卡罗抽样的高效适应性来推导贝叶斯推理算法。模式理论的使用允许弥合原始信号和高级依赖于领域的语义之间的差距，并有助于发现可能代表相同底层事件的大组视听事件。这项工作结合了计算机视觉中的知觉组织、听觉信号的计算分析、模式理论和本体结构的先前发展的想法。本文提出的方法适用于许多部署音视频传感器的场景，包括视频的音频标注问题、电话会议中的说话人跟踪问题、远程监控中的多目标分离问题。更广泛的影响活动包括开发教学模块、对学生进行创新和创业培训，以及向社区宣传调查结果。

It is natural that events of interest in observed scenes manifest themselves across multiple sensing modalities - vision, hearing, smell, etc. The remarkable perceptions of audio-video signals by natural systems, such as humans, also points to superiority of inferences drawn across modalities. It, therefore, seems natural to enhance performance of automated systems by using joint, cross-modal statistical inferences. However, the detection, organization, and understanding of cues and events in real-world scenarios are difficult tasks. This project seeks to develop a pattern-theoretic framework for achieving these goals. The main research items are: (1) development of mathematical quantities to represent audio and visual events and their spatiotemporal relations, (2) use domain-specific ontologies to impose semantic structure and to incorporate prior knowledge, and (3) derive algorithms for Bayesian inferences using efficient adaptations of Markov Chain Monte Carlo sampling. The use of pattern theory allows bridging of gaps between raw signals and high-level, domain-dependent semantics, and helps discovers large groups of audio-visual events likely to represent the same underlying event. This effort combines ideas from perceptual organization in computer vision, computational analysis of auditory signals, pattern theory, and prior developments in ontological structures. The methods developed here are applicable to many scenarios that deploy audio and video sensors, including problems of audio annotations of videos, speaker tracking in teleconferencing, and separation of multiple objects in remote surveillance. Broader impact activities involve the development of teaching modules, innovation and entrepreneurial training of the students, and communication of the findings to the community.