A Study of Dynamic Audiovisual Integration based on High Speed Vision

在本研究中，我们构建了一种动态视听集成理论，通过匹配视觉传感器和听觉传感器的采样和信号处理速率，在人类可听频域（20 ~ 20kHz）的信号水平上集成视觉传感器和听觉传感器信息。基于视觉信息和听觉信息共同包含的振动动态，证明了动态视听集成的有效性。具体而言，我们研究了以下问题；A)视觉听觉（VA）信号处理，将视觉信息实时转化为声音信息；B)逐像素声源检测，提取视觉信息与听觉信息之间的共同动态关系；C)基于声速与光速之差的声源距离/速度测量。在帧率为kHz的高速视觉系统上实现了这些问题，验证了动态视听集成的基本功能。

In this research, we constructed a theory for dynamic audiovisual integration which can integrate both vision sensor information and auditory sensor information at signal level in human audible frequency domain (20〜20kHz) by matching sampling and signal processing rates of vision sensor and auditory sensor. We also showed the effectiveness of dynamic audiovisual integration based on the vibrated dynamics which is contained in both visual information and auditory information commonly.Concretely, we studied the following issues ; A)VA (Visual Auditorizaition) signal processing, in which visual information is converted into sound information at real-time, B)Pixel-wise Sound Source Detection by extracting the common dynamics between visual information and auditory information, C)Sound source distance / velocity measurement based on the difference between sound speed and light speed. These issues were implemented on the high speed vision system whose frame rate is in kHz order, and we verified the fundamental function of dynamic audiovisual integration.