Sonic Visual Interface

在这项研究中，我们研究了获取3D视觉信息和3D声音信息之间的一致性，以及利用声音信息来表示视觉信息的方法。首先分析了3D图像空间、3D声音空间和3D听觉感知空间之间的关系。如果摄像机未校准，则从图像重建的3D空间具有3D投影模糊性。如果3D声音发生器未校准，则3D声音空间具有3D投影模糊性。此外，我们在人类的听觉感知中也存在3D仿射模糊。因此，3D图像空间、3D声音空间和3D听觉空间之间的关系可以用3D投影变换来描述。利用这些性质，我们提出了一种利用3D声音来表示3D视觉形状和位置的方法，并将该方法应用于混合现实应用中获取视觉和听觉信息的几何一致性。特别是，我们表明，通过使用多个摄像机的相互投影，我们可以比现有的方法更准确地计算3D视觉信息。我们还将提出的方法应用于多用户混合现实系统，并表明通过在每个用户的相机坐标中恢复3D信息，我们可以为每个人生成3D声音信息。将提出的方法应用于虚拟格斗系统和虚拟乐器中的虚拟图像和虚拟声音的生成

In this research, we investigated methods for taking consistency between 3D visual information and 3D sound information, and for representing visual information by using sound information. We first analyzed the relationship between 3D image space, 3D sound space and 3D auditory perception space. If cameras are uncalibrated, 3D space reconstructed from images has 3D projective ambiguity. IF 3D sound generators are uncalibrated, 3D sound space has 3D projective ambiguity. Also, We have 3D affine ambiguity in human auditory perception. Thus, the relationship between 3D image space, 3D sound space and 3D auditory space can be described by 3D projective transformations. By using these properties, we proposed a method for representing 3D visual shapes and positions by using 3D sound.We next applied the method for taking geometric consistency of visual and auditory information in mixed reality applications. In particular, we showed that by using the mutual projection of multiple cameras, we can compute 3D visual information much more accurately than the existing method. We also applied the proposed method for mixed reality systems of multiple users, and showed that by recovering 3D information in each user's camera coordinates, we can generate 3D sound information for each person. We applied the proposed method for generating virtual images and virtual sounds in virtual fighting systems and in virtual music instruments