Audiovisual Speaker Conversion: Jointly and Simultaneously Transforming Facial Expression and Acoustic Characteristics
Audiovisual Speaker Conversion: Jointly and Simultaneously Transforming Facial Expression and Acoustic Characteristics
复制标题
DOI：
10.1109/icassp.2019.8683872
复制
发表时间：
2018-10
期刊：
影响因子：
--
通讯作者：
Fuming Fang;Xin Wang;J. Yamagishi;I. Echizen
中科院分区：
文献类型：
--
作者：
Fuming Fang;Xin Wang;J. Yamagishi;I. Echizen
提出了一种视听说话人转换方法，用于同时将源说话人的面部表情和语音转换为目标说话人的面部表情和语音。将面部和声学特征一起变换使得转换后的语音和面部表情高度相关，并且生成的目标说话者看起来和听起来自然。它使用三个神经网络：融合并变换面部特征和声学特征的转换网络、从经转换的面部特征和声学特征两者产生波形的波形生成网络、以及也基于经转换的特征两者输出RGB面部图像的图像重构网络。实验结果表明，使用情感视听数据库，该方法实现了显着更高的自然度相比，分别变换的声学和面部特征。
An audiovisual speaker conversion method is presented for simultaneously transforming the facial expressions and voice of a source speaker into those of a target speaker. Transforming the facial and acoustic features together makes it possible for the converted voice and facial expressions to be highly correlated and for the generated target speaker to appear and sound natural. It uses three neural networks: a conversion network that fuses and transforms the facial and acoustic features, a waveform generation network that produces the waveform from both the converted facial and acoustic features, and an image reconstruction network that outputs an RGB facial image also based on both the converted features. The results of experiments using an emotional audiovisual database showed that the proposed method achieved significantly higher naturalness compared with one that separately transformed acoustic and facial features.