Augmented speech communication using multi-modal signals with real-time, low-latency voice conversion

使用具有实时、低延迟语音转换的多模信号的增强语音通信

• 批准号: 22KJ1519
• 负责人: HUANG WENCHIN
• 金额: $ 1.41万
• 依托单位: Nagoya University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for JSPS Fellows
• 财政年份: 2023
• 资助国家: 日本
• 起止时间: 2023-03-08 至 2024-03-31
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-22KJ1519/
• 关键词: voice conversion; self-supervised learning; dysarthria; electrolaryngeal speech

The purpose of this research is to apply voice conversion (VC) to realize an interactive speech production paradigm for real-world applications, with the help of multimodal signals and real-time processing techniques. In the second year, the applicant focused on three aspects.(1) Continued improvement on fundamental VC techniques, specifically self-supervised speech representation (S3R)-based VC, an emerging trend which reduces training data requirements. The applicant kept on updating S3PRL-VC, an open-source toolkit for researchers to evaluate S3R models for VC, and published the latest experimental results in the IEEE Journal of Selected Topics in Signal Processing.(2) Foreign accent conversion, a task that helps reduce foreign accents for efficient communication. A paper that provides an unified evaluation of current approaches and identifies unsolved problems is submitted to an international conference and currently under review.(3) Singing voice conversion, a fundamental technique that has the potential to augment the communication ability of human. The applicant is running a scientific event named the Singing Voice Conversion Challenge 2023, which aims to provide an unified experimental setting including task and dataset, in order to attract researchers world-wide to look into this problem and explore the limitation of the state-of-the-art techniques.

本研究的目的是应用语音转换（VC），以实现一个交互式语音生产的范例，为现实世界的应用程序，与多模态信号和实时处理技术的帮助。第二年，申请人重点关注了三个方面。(1)基本VC技术的持续改进，特别是基于自监督语音表示（S3 R）的VC，这是一种新兴趋势，可降低训练数据要求。申请人不断更新S3 PRL-VC，这是一个供研究人员评估VC的S3 R模型的开源工具包，并在IEEE Journal of Selected Topics in Signal Processing上发表了最新的实验结果。(2)外国口音转换，这是一项有助于减少外国口音以提高沟通效率的任务。向一次国际会议提交了一份文件，对目前的做法进行了统一评价，并查明了尚未解决的问题，目前正在审查该文件。(3)歌唱声转换技术是一项有潜力提高人类交流能力的基础技术。申请人正在举办一项名为“2023年歌声转换挑战赛”的科学活动，旨在提供包括任务和数据集在内的统一实验设置，以吸引世界各地的研究人员研究这一问题并探索最新技术的局限性。

项目成果

A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion

• DOI: 10.21437/interspeech.2021-208
• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Wen-Chin Huang;Kazuhiro Kobayashi;Yu-Huai Peng;Ching-Feng Liu;Yu Tsao;Hsin-Min Wang;T. Toda

CRANK: an Open-Source Software for Nonparallel Voice Conversion based on Vetor-Quantized Variational Autoencoder

CRANK：基于矢量量化变分自动编码器的非并行语音转换开源软件

• 发表时间: 2021
• 作者: Kazuhiro Kobayashi;Wen-Chin Huang;Yi-Chiao Wu;Patrick Tobing;Tomoki Hayashi;and Tomoki Toda
• 通讯作者: and Tomoki Toda

On Prosody Modeling for ASR+TTS Based Voice Conversion

• DOI: 10.1109/asru51503.2021.9688010
• 发表时间: 2021-07
• 期刊: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
• 作者: Wen-Chin Huang;Tomoki Hayashi;Xinjian Li;Shinji Watanabe;T. Toda

S3PRL-VC: Open-Source Voice Conversion Framework with Self-Supervised Speech Representations

• DOI: 10.1109/icassp43922.2022.9746430
• 发表时间: 2021-10
• 期刊: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: Wen-Chin Huang;Shu-Wen Yang;Tomoki Hayashi;Hung-yi Lee;Shinji Watanabe;T. Toda