A Study of Dynamic Audiovisual Integration based on High Speed Vision

在本研究中，我们建立了一个动态视听融合理论，它可以通过匹配视觉传感器和听觉传感器的采样率和信号处理率，在人类可听频域（20 ~ 20 kHz）的信号水平上融合视觉传感器和听觉传感器的信息。同时，我们还研究了基于视觉信息和听觉信息中普遍存在的振动动力学的动态视听整合的有效性。A）VA（视觉听觉）信号处理，其中视觉信息被实时转换成声音信息，B）通过提取视觉信息和听觉信息之间的共同动态的逐像素声源检测，C）基于声速和光速之间的差异的声源距离/速度测量。这些问题在帧速率为kHz量级的高速视觉系统上得到了实现，验证了动态视听融合的基本功能。

In this research, we constructed a theory for dynamic audiovisual integration which can integrate both vision sensor information and auditory sensor information at signal level in human audible frequency domain (20〜20kHz) by matching sampling and signal processing rates of vision sensor and auditory sensor. We also showed the effectiveness of dynamic audiovisual integration based on the vibrated dynamics which is contained in both visual information and auditory information commonly.Concretely, we studied the following issues ; A)VA (Visual Auditorizaition) signal processing, in which visual information is converted into sound information at real-time, B)Pixel-wise Sound Source Detection by extracting the common dynamics between visual information and auditory information, C)Sound source distance / velocity measurement based on the difference between sound speed and light speed. These issues were implemented on the high speed vision system whose frame rate is in kHz order, and we verified the fundamental function of dynamic audiovisual integration.