Unsupervised audio-visual geometry calibration of distributed microphone arrays

该项目的总体目标是简化视听传感器网络的安装。这是通过开发从混响语音输入自动确定分布式麦克风阵列位置的算法来实现的。然后，建立了声传感器网络感知到的事件与另一模态传感器网络感知到的事件之间的对应关系。另一种模式是由多摄像机网络给出的，其中假定摄像机位置是已知的。一旦检测到对应关系，声学传感器网络的相对几何形状就可以映射到给定的摄像机网络坐标系。在第一个项目阶段，我们开发了二维麦克风阵列位置自校准算法，其特点是不受混响影响，并且可以在语音输入上运行，不需要人工校准信号。所要求的项目扩展致力于以下目标：1)将在声学传感器网络的相对几何形状与由多摄像机系统给出的全球坐标系统之间建立对应关系。这将通过将从声学子系统获得的说话人的轨迹映射到视觉相关者（即面孔或人物）的轨迹来实现。2)开发的几何自校准系统将嵌入到环境通信系统中进行评估。在环境通信系统中，声波束形成器提取语音信号，而有源摄像机系统跟踪说话人。在该试验台中，可以在实时环境和实际环境条件下研究用户与系统之间的交互，最终实现校准系统的优化。

The overall goal of the project is to simplify the installation of audio-visual sensor networks. This is achieved by developing algorithms for automatically determining the position of the distributed microphone arrays from reverberant speech input. Then correspondences between events sensed by the acoustic sensor network and those sensed by a sensor network of another modality are established. This other modality is given by a multi-camera-network, where the camera positions are assumed to be known. Once the correspondences have been detected the relative geometries of the acoustic sensor network can be mapped to the given coordinate system of the camera network.In the first project phase we have developed algorithms for microphone array posiltion self calibration in 2D, which are characterized by immunity to reverberation and which can operate on speech input, not requiring artificial calibration signals.The requested project extension is devoted to the following objectives:1) A correspondence will be developed between the relative geometries of the acoustic sensor network and a global coordinate system, which is given by a multi-camera system. This shall be achieved by mapping trajectories of speakers obtained from the acoustic subsystem to trajectories of the visual correlates, i.e., faces or persons.2) The developed geometry self calibration system will be embedded in an ambient communication system for evaluation purposes.In the ambient communication system an acoustic beamformer extracts the speech signal, while the active camera system tracks the speaker. In this test bed the interaction between user and system can be studied in a real-time environment and under realistic environmental conditions, eventually leading to an optimization of the calibration system.