Augmented speech communication using multi-modal signals with real-time, low-latency voice conversion

使用具有实时、低延迟语音转换的多模信号的增强语音通信

• 批准号: 22KJ1519
• 负责人: HUANG WENCHIN
• 金额: $ 1.41万
• 依托单位: Nagoya University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for JSPS Fellows
• 财政年份: 2023
• 资助国家: 日本
• 起止时间: 2023-03-08 至 2024-03-31
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-22KJ1519/
• 关键词: voice conversion; self-supervised learning; dysarthria; electrolaryngeal speech

The purpose of this research is to apply voice conversion (VC) to realize an interactive speech production paradigm for real-world applications, with the help of multimodal signals and real-time processing techniques. In the second year, the applicant focused on three aspects.(1) Continued improvement on fundamental VC techniques, specifically self-supervised speech representation (S3R)-based VC, an emerging trend which reduces training data requirements. The applicant kept on updating S3PRL-VC, an open-source toolkit for researchers to evaluate S3R models for VC, and published the latest experimental results in the IEEE Journal of Selected Topics in Signal Processing.(2) Foreign accent conversion, a task that helps reduce foreign accents for efficient communication. A paper that provides an unified evaluation of current approaches and identifies unsolved problems is submitted to an international conference and currently under review.(3) Singing voice conversion, a fundamental technique that has the potential to augment the communication ability of human. The applicant is running a scientific event named the Singing Voice Conversion Challenge 2023, which aims to provide an unified experimental setting including task and dataset, in order to attract researchers world-wide to look into this problem and explore the limitation of the state-of-the-art techniques.

本研究的目的是借助多模态信号和实时处理技术，应用语音转换（VC）来实现现实世界应用的交互式语音生成范例。第二年，申请人重点关注三个方面。（1）基础VC技术的持续改进，特别是基于自监督语音表示（S3R）的VC，这是一种减少训练数据需求的新兴趋势。申请人不断更新开源工具包S3PRL-VC，供研究人员评估VC的S3R模型，并将最新实验结果发表在IEEE Journal of Selected Topics in Signal Processing上。（2）外国口音转换，一项有助于减少外国口音以实现高效沟通的任务。一篇对当前方法进行统一评估并确定未解决问题的论文已提交给国际会议，目前正在审查中。(3) 歌声转换，一种有潜力增强人类交流能力的基本技术。申请人正在举办一项名为“2023年歌声转换挑战赛”的科学活动，旨在提供一个包括任务和数据集在内的统一实验环境，以吸引世界各地的研究人员研究这个问题并探索最先进技术的局限性。

项目成果

A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion

• DOI: 10.21437/interspeech.2021-208
• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Wen-Chin Huang;Kazuhiro Kobayashi;Yu-Huai Peng;Ching-Feng Liu;Yu Tsao;Hsin-Min Wang;T. Toda

CRANK: an Open-Source Software for Nonparallel Voice Conversion based on Vetor-Quantized Variational Autoencoder

CRANK：基于矢量量化变分自动编码器的非并行语音转换开源软件

• 发表时间: 2021
• 作者: Kazuhiro Kobayashi;Wen-Chin Huang;Yi-Chiao Wu;Patrick Tobing;Tomoki Hayashi;and Tomoki Toda
• 通讯作者: and Tomoki Toda

On Prosody Modeling for ASR+TTS Based Voice Conversion

• DOI: 10.1109/asru51503.2021.9688010
• 发表时间: 2021-07
• 期刊: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
• 作者: Wen-Chin Huang;Tomoki Hayashi;Xinjian Li;Shinji Watanabe;T. Toda

S3PRL-VC: Open-Source Voice Conversion Framework with Self-Supervised Speech Representations

• DOI: 10.1109/icassp43922.2022.9746430
• 发表时间: 2021-10
• 期刊: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: Wen-Chin Huang;Shu-Wen Yang;Tomoki Hayashi;Hung-yi Lee;Shinji Watanabe;T. Toda