ITR: Analysis of Complex Audio-Visual Events Using Spatially Distributed Sensors

我们建议开发一个全面的框架，用于联合分析从空间分布的麦克风和摄像机获得的视听信号。我们希望视听传感问题的解决方案能够扩展到任意数量的摄像头和麦克风，并能够应对具有挑战性的环境，在这种环境中，有多个语音和非语音声源，以及多个运动的人和物体。最近，在一个环境中部署数十个甚至数百个摄像头和麦克风变得相对便宜。许多应用程序可以受益于在两种模式下进行感知的能力。在两个层次上可以进行联合视听分析。在信号水平上，挑战是开发能够捕捉联合信号中丰富的依赖结构的表示，并成功地处理诸如可变采样率和不同提示之间的时延等问题。在空间层面上的挑战是补偿传感器位置引起的失真，并在传感器之间汇集信息以恢复关于空间环境的3-D信息。对于许多应用来说，如果解决方法是自校准是非常理想的，并且不需要每次添加新传感器或移动或更换旧传感器时都需要大量的人工校准过程。消除了人工校准的负担，也使得利用例如来自可穿戴麦克风和摄像机的自组织传感器网络成为可能。我们建议解决以下四个研究主题：1.信号电平融合的表示和学习方法。融合空间分布的视听数据的体积技术。分布式麦克风-摄像机系统的自标定视听的应用。例如，这项建议包括在嘴唇和面部分析方面的大量工作，以改善语音通信。

We propose to develop a comprehensive framework for the joint analysis of audio-visual signals obtainedfrom spatially distributed microphones and cameras. We desire solutions to the audio-visual sensing problem that will scale to an arbitrary number of cameras and microphones and can address challenging environments in which there are multiple speech and nonspeech sound sources and multiple moving people and objects. Recently it has become relatively inexpensive to deploy tens or even hundreds of cameras and microphones in an environment. Many applications could benefit from ability to sense in both modalities.There are two levels at which joint audio-visual analysis can take place. At the signal level, the challengeis to develop representations that capture the rich dependency structure in the joint signal and deal success-fully issues such as variable sampling rates and varying temporal delays between cues. At the spatial level the challenge is to compensate for the distortions introduced by the sensor location and pool information across sensors to recover 3-D information about the spatial environment.For many applications, it is highly desirable if the solution method is self-calibrating, and does notrequire an extensive manual calibration process every time a new sensor is added or an old sensor is movedor replaced. Removing the burden of manual calibration also makes it possible to exploit ad hoc sensornetworks which could arise, for example, from wearable microphones and cameras.We propose to address the following four research topics:1. Representations and learning methods for signal level fusion.2. Volumetric techniques for fusing spatially distributed audio-visual data.3. Self-calibration of distributed microphone-camera systems4. Applications of audio-visual sensing.For example, this proposal includes considerable work on lip and facial analysis to improve voicecommunications.