ITR: Analysis of Complex Audio-Visual Events Using Spatially Distributed Sensors

我们建议开发一个全面的框架，用于联合分析从空间分布的麦克风和摄像机获得的视听信号。我们希望解决方案的视听传感问题，将规模到任意数量的摄像头和麦克风，并可以解决具有挑战性的环境中，有多个语音和非语音声源和多个移动的人和物体。 最近，在一个环境中部署数十甚至数百个摄像机和麦克风变得相对便宜。许多应用程序可以受益于两种模式的感知能力。有两个层次的联合视听分析可以发生。在信号级，挑战是开发表示，捕捉丰富的依赖结构的联合信号和成功地处理问题，如可变的采样率和不同的时间延迟之间的线索。在空间层次上，挑战在于如何补偿由传感器位置引入的失真，并将传感器之间的信息汇集起来，以恢复空间环境的三维信息。对于许多应用，如果解决方法是自校准，并且不需要每次添加新传感器或移动或更换旧传感器时都进行大量的手动校准过程，则是非常理想的。去除手动校准的负担也使得利用ad hoc sensornetworks成为可能，例如，从可穿戴麦克风和摄像头。我们提出了以下四个研究课题：1。信号级融合的表示和学习方法.融合空间分布的视听数据的体积技术.分布式麦克风相机系统的自校准4.视听感知的应用。例如，这项建议包括大量的唇和面部分析工作，以提高识别能力。

We propose to develop a comprehensive framework for the joint analysis of audio-visual signals obtainedfrom spatially distributed microphones and cameras. We desire solutions to the audio-visual sensing problem that will scale to an arbitrary number of cameras and microphones and can address challenging environments in which there are multiple speech and nonspeech sound sources and multiple moving people and objects. Recently it has become relatively inexpensive to deploy tens or even hundreds of cameras and microphones in an environment. Many applications could benefit from ability to sense in both modalities.There are two levels at which joint audio-visual analysis can take place. At the signal level, the challengeis to develop representations that capture the rich dependency structure in the joint signal and deal success-fully issues such as variable sampling rates and varying temporal delays between cues. At the spatial level the challenge is to compensate for the distortions introduced by the sensor location and pool information across sensors to recover 3-D information about the spatial environment.For many applications, it is highly desirable if the solution method is self-calibrating, and does notrequire an extensive manual calibration process every time a new sensor is added or an old sensor is movedor replaced. Removing the burden of manual calibration also makes it possible to exploit ad hoc sensornetworks which could arise, for example, from wearable microphones and cameras.We propose to address the following four research topics:1. Representations and learning methods for signal level fusion.2. Volumetric techniques for fusing spatially distributed audio-visual data.3. Self-calibration of distributed microphone-camera systems4. Applications of audio-visual sensing.For example, this proposal includes considerable work on lip and facial analysis to improve voicecommunications.