ITR: Analysis of Complex Audio-Visual Events Using Spatially Distributed Sensors

我们建议开发一个综合框架，用于联合分析从空间分布的麦克风和摄像机获得的视听信号。我们希望解决视听传感问题的解决方案能够扩展到任意数量的摄像机和麦克风，并能够解决具有挑战性的环境，其中存在多个语音和非语音声源以及多个移动的人和物体。最近，在一个环境中部署数十甚至数百个摄像头和麦克风已经变得相对便宜。许多应用都可以从两种方式的感知能力中受益。可以在两个层次上进行联合视听分析。在信号层面，挑战在于开发表征，以捕获联合信号中丰富的依赖结构，并成功处理诸如可变采样率和线索之间的不同时间延迟等问题。在空间层面上，挑战在于补偿传感器位置带来的失真，并在传感器之间汇集信息以恢复有关空间环境的三维信息。对于许多应用来说，如果解决方法是自校准的，并且每次添加新传感器或移动或更换旧传感器时不需要广泛的手动校准过程，这是非常可取的。消除手动校准的负担也使利用可穿戴麦克风和摄像头等可能出现的特别传感器网络成为可能。我们建议解决以下四个研究课题：1。信号级融合的表示与学习方法融合空间分布视听数据的体积技术。分布式麦克风-摄像机系统的自标定视听传感的应用。例如，该提案包括大量的唇和面部分析工作，以改善语音通信。

We propose to develop a comprehensive framework for the joint analysis of audio-visual signals obtainedfrom spatially distributed microphones and cameras. We desire solutions to the audio-visual sensing problem that will scale to an arbitrary number of cameras and microphones and can address challenging environments in which there are multiple speech and nonspeech sound sources and multiple moving people and objects. Recently it has become relatively inexpensive to deploy tens or even hundreds of cameras and microphones in an environment. Many applications could benefit from ability to sense in both modalities.There are two levels at which joint audio-visual analysis can take place. At the signal level, the challengeis to develop representations that capture the rich dependency structure in the joint signal and deal success-fully issues such as variable sampling rates and varying temporal delays between cues. At the spatial level the challenge is to compensate for the distortions introduced by the sensor location and pool information across sensors to recover 3-D information about the spatial environment.For many applications, it is highly desirable if the solution method is self-calibrating, and does notrequire an extensive manual calibration process every time a new sensor is added or an old sensor is movedor replaced. Removing the burden of manual calibration also makes it possible to exploit ad hoc sensornetworks which could arise, for example, from wearable microphones and cameras.We propose to address the following four research topics:1. Representations and learning methods for signal level fusion.2. Volumetric techniques for fusing spatially distributed audio-visual data.3. Self-calibration of distributed microphone-camera systems4. Applications of audio-visual sensing.For example, this proposal includes considerable work on lip and facial analysis to improve voicecommunications.