A Study of Dynamic Audiovisual Integration based on High Speed Vision

在本研究中，我们通过匹配视觉传感器和听觉传感器的采样和信号处理速率，构建了一种能够在人类可听频域(20~20 kHz)的信号级上同时整合视觉传感器信息和听觉传感器信息的动态视听整合理论。具体研究了以下几个问题：A)将视觉信息实时转换为声音信息的VA(视觉审计)信号处理；B)通过提取视觉信息和听觉信息之间的共同动力学信息进行像素级声源检测；C)基于声速和光速差的声源距离/速度测量。在帧速率为KHz量级的高速视觉系统上实现了上述问题，验证了动态视听融合的基本功能。

In this research, we constructed a theory for dynamic audiovisual integration which can integrate both vision sensor information and auditory sensor information at signal level in human audible frequency domain (20〜20kHz) by matching sampling and signal processing rates of vision sensor and auditory sensor. We also showed the effectiveness of dynamic audiovisual integration based on the vibrated dynamics which is contained in both visual information and auditory information commonly.Concretely, we studied the following issues ; A)VA (Visual Auditorizaition) signal processing, in which visual information is converted into sound information at real-time, B)Pixel-wise Sound Source Detection by extracting the common dynamics between visual information and auditory information, C)Sound source distance / velocity measurement based on the difference between sound speed and light speed. These issues were implemented on the high speed vision system whose frame rate is in kHz order, and we verified the fundamental function of dynamic audiovisual integration.