Cocktail party effect procesor using audio-visual sensor fusion technique

“鸡尾酒会效应”一词是指从其他环境噪音中提取重要信号的生物学特殊能力。本文主要研究鸡尾酒会效应，以期构建一种新型的传感系统。本文提出了用于传感器融合的二维逻辑元件阵列，即在时空坐标上集成音频信号和序列图像的二维传感器融合系统。研究总结如下。今年，作为初步系统制作的8 × 8阵列的小型原型系统扩展为24 × 16阵列的大型系统。实验结果证明了新系统的优越性，特别是在多路复用数据的情况下。今年还开发了自动音高跟踪算法，作为声源定位的初步步骤。通过实验验证了该算法对源定位子系统的改进。另一个子系统是图像处理部分。在这一部分中，基于独立分量分析理论，提出了一种基于特征点的新算法。利用这类特征点，对立体视觉子系统进行了实验验证。结果表明，二维逻辑单元的概念在时空坐标下的传感器融合中具有优势。在未来，系统将进行修订，以便更定量地评估算法的效率。

The term "Cocktail party effect" means biological special ability to extract an important signal from other ambient noises. In this research, the cocktail party effect is focused on in order to construct a new type of sensing system. Two dimensional array of logic elements for sensor fusion, i.e., two dimensional sensor fusion system which integrate audio signals and sequential images on time-space coordinates, is proposed in this research. Summaries of the research are as followas.A small prototype system wiht an array of 8x8, which had been made as preliminary system, was extended to a larger system with an array of 24x16 inthis year. Results of experiments using the revised system demonstraed the advantages of the concept of the new system, especially in the case of multiplexed data.Automatic pitch tracking algorithm was also developed in this year, which was used as a preliminary step of sound source localization. Using this algorithm, improvements in the source localization subsystem were demonstrated experimentally.The other subsystem is an image processing part. In this part, a new algorithm using feature points was developed, which wa based on the theory of independent component analysis. Using this type of feature points, improvements in the stereo vision subsystem were demonstrated experimentally.As the consequence of these results, the advantages of the concept of two dimensional logic elements for sensor fusion on time-space coordinates were shown. In the future, the system will be revised in order to evaluate the efficiency of the algorithm more quantitatively.