Cocktail party effect procesor using audio-visual sensor fusion technique

“鸡尾酒会效应”指的是从其他环境噪音中提取重要信号的生物特殊能力。本研究以鸡尾酒会效应为研究对象，旨在构建一种新型的传感系统。提出了一种用于传感器融合的二维逻辑单元阵列，即在时空坐标上集成音频信号和序列图像的二维传感器融合系统。主要研究内容如下：1.在今年将一个以为阵列的小型原型系统扩展到了一个24x16阵列的较大系统。使用改进后的系统进行的实验结果表明，新系统的概念具有优势，特别是在数据多路传输的情况下。今年还开发了自动基音跟踪算法，作为声源定位的初步步骤。利用该算法对信源定位子系统的改进进行了实验验证。另一个子系统是图像处理部分。在这一部分中，基于独立分量分析理论，提出了一种新的利用特征点的算法。利用这种类型的特征点，实验证明了立体视觉子系统的改进，从而显示了二维逻辑元素概念在时空坐标上用于传感器融合的优势。未来将对该系统进行修改，以便更定量地评估算法的效率。

The term "Cocktail party effect" means biological special ability to extract an important signal from other ambient noises. In this research, the cocktail party effect is focused on in order to construct a new type of sensing system. Two dimensional array of logic elements for sensor fusion, i.e., two dimensional sensor fusion system which integrate audio signals and sequential images on time-space coordinates, is proposed in this research. Summaries of the research are as followas.A small prototype system wiht an array of 8x8, which had been made as preliminary system, was extended to a larger system with an array of 24x16 inthis year. Results of experiments using the revised system demonstraed the advantages of the concept of the new system, especially in the case of multiplexed data.Automatic pitch tracking algorithm was also developed in this year, which was used as a preliminary step of sound source localization. Using this algorithm, improvements in the source localization subsystem were demonstrated experimentally.The other subsystem is an image processing part. In this part, a new algorithm using feature points was developed, which wa based on the theory of independent component analysis. Using this type of feature points, improvements in the stereo vision subsystem were demonstrated experimentally.As the consequence of these results, the advantages of the concept of two dimensional logic elements for sensor fusion on time-space coordinates were shown. In the future, the system will be revised in order to evaluate the efficiency of the algorithm more quantitatively.