Sonic Visual Interface

点击翻译按钮获取中文摘要

In this research, we investigated methods for taking consistency between 3D visual information and 3D sound information, and for representing visual information by using sound information. We first analyzed the relationship between 3D image space, 3D sound space and 3D auditory perception space. If cameras are uncalibrated, 3D space reconstructed from images has 3D projective ambiguity. IF 3D sound generators are uncalibrated, 3D sound space has 3D projective ambiguity. Also, We have 3D affine ambiguity in human auditory perception. Thus, the relationship between 3D image space, 3D sound space and 3D auditory space can be described by 3D projective transformations. By using these properties, we proposed a method for representing 3D visual shapes and positions by using 3D sound.We next applied the method for taking geometric consistency of visual and auditory information in mixed reality applications. In particular, we showed that by using the mutual projection of multiple cameras, we can compute 3D visual information much more accurately than the existing method. We also applied the proposed method for mixed reality systems of multiple users, and showed that by recovering 3D information in each user's camera coordinates, we can generate 3D sound information for each person. We applied the proposed method for generating virtual images and virtual sounds in virtual fighting systems and in virtual music instruments