Visual Onoma-to-Wave: Environmental Sound Synthesis from Visual Onomatopoeias and Sound-Source Images
Visual Onoma-to-Wave: Environmental Sound Synthesis from Visual Onomatopoeias and Sound-Source Images
复制标题
DOI：
10.1109/icassp49357.2023.10096517
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Hien Ohnaka;Shinnosuke Takamichi;Keisuke Imoto;Yuki Okamoto;Kazuki Fujii;H. Saruwatari
中科院分区：
文献类型：
--
作者：
Hien Ohnaka;Shinnosuke Takamichi;Keisuke Imoto;Yuki Okamoto;Kazuki Fujii;H. Saruwatari
我们提出了一种方法来合成环境声音从视觉上表示拟声词和声源。拟声词是一个模仿声音结构的词，即，声音的文本表示。从这个角度来看，拟声词到波已经被提出来合成环境声音从所需的拟声词文本。拟声词还有另一种表现形式：漫画、广告和虚拟现实中声音的视觉-文本表现形式。视觉拟声词（拟声词的视觉文本）包含文本中不存在的丰富信息，例如图像的长短持续时间，因此使用这种表示法有望合成不同的声音。因此，我们提出了视觉拟声到波的环境声音合成从视觉拟声。该方法可以将视觉文本和声源图像的视觉概念转换为合成声音。我们还提出了一个数据增强方法，重点是重复的拟声词，以提高我们的方法的性能。实验结果表明，该方法可以从视觉文本和声源图像合成不同的环境声音。
We propose a method for synthesizing environmental sounds from visually represented onomatopoeias and sound sources. An onomatopoeia is a word that imitates a sound structure, i.e., the text representation of sound. From this perspective, onoma-to-wave has been proposed to synthesize environmental sounds from the desired onomatopoeia texts. Onomatopoeias have another representation: visual-text representations of sounds in comics, advertisements, and virtual reality. A visual onomatopoeia (visual text of onomatopoeia) contains rich information that is not present in the text, such as a long-short duration of the image, so the use of this representation is expected to synthesize diverse sounds. Therefore, we propose visual onoma-to-wave for environmental sound synthesis from visual onomatopoeia. The method can transfer visual concepts of the visual text and sound-source image to the synthesized sound. We also propose a data augmentation method focusing on the repetition of onomatopoeias to enhance the performance of our method. An experimental evaluation shows that the methods can synthesize diverse environmental sounds from visual text and sound-source images.