Augmented speech communication using multi-modal signals with real-time, low-latency voice conversion

使用具有实时、低延迟语音转换的多模信号的增强语音通信

• 批准号: 22KJ1519
• 负责人: HUANG WENCHIN
• 金额: $ 1.41万
• 依托单位: Nagoya University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for JSPS Fellows
• 财政年份: 2023
• 资助国家: 日本
• 起止时间: 2023-03-08 至 2024-03-31
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-22KJ1519/
• 关键词: voice conversion; self-supervised learning; dysarthria; electrolaryngeal speech

The purpose of this research is to apply voice conversion (VC) to realize an interactive speech production paradigm for real-world applications, with the help of multimodal signals and real-time processing techniques. In the second year, the applicant focused on three aspects.(1) Continued improvement on fundamental VC techniques, specifically self-supervised speech representation (S3R)-based VC, an emerging trend which reduces training data requirements. The applicant kept on updating S3PRL-VC, an open-source toolkit for researchers to evaluate S3R models for VC, and published the latest experimental results in the IEEE Journal of Selected Topics in Signal Processing.(2) Foreign accent conversion, a task that helps reduce foreign accents for efficient communication. A paper that provides an unified evaluation of current approaches and identifies unsolved problems is submitted to an international conference and currently under review.(3) Singing voice conversion, a fundamental technique that has the potential to augment the communication ability of human. The applicant is running a scientific event named the Singing Voice Conversion Challenge 2023, which aims to provide an unified experimental setting including task and dataset, in order to attract researchers world-wide to look into this problem and explore the limitation of the state-of-the-art techniques.

本研究的目的是在多模态信号和实时处理技术的帮助下，应用语音转换（VC）来实现现实世界应用的交互式语音生产范式。在第二年，申请人着重于三个方面。(1)持续改进基础VC技术，特别是基于自监督语音表示（S3R）的VC，这是一种降低训练数据需求的新兴趋势。申请人持续更新用于研究人员评估VC的S3R模型的开源工具包S3PRL-VC，并在IEEE Journal of Selected Topics in Signal Processing上发表最新实验结果。(2)外国口音转换，减少外国口音，提高沟通效率。向一次国际会议提交了一份文件，该文件对目前的方法进行了统一评价，并确定了尚未解决的问题，目前正在审查中。(3)歌唱声音转换是一种具有增强人类交流能力潜力的基本技术。申请人正在举办名为“歌唱声音转换挑战2023”的科学活动，旨在提供包括任务和数据集在内的统一实验环境，以吸引世界各地的研究人员研究这一问题，并探索最先进技术的局限性。

项目成果

A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion

• DOI: 10.21437/interspeech.2021-208
• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Wen-Chin Huang;Kazuhiro Kobayashi;Yu-Huai Peng;Ching-Feng Liu;Yu Tsao;Hsin-Min Wang;T. Toda

CRANK: an Open-Source Software for Nonparallel Voice Conversion based on Vetor-Quantized Variational Autoencoder

CRANK：基于矢量量化变分自动编码器的非并行语音转换开源软件

• 发表时间: 2021
• 作者: Kazuhiro Kobayashi;Wen-Chin Huang;Yi-Chiao Wu;Patrick Tobing;Tomoki Hayashi;and Tomoki Toda
• 通讯作者: and Tomoki Toda

On Prosody Modeling for ASR+TTS Based Voice Conversion

• DOI: 10.1109/asru51503.2021.9688010
• 发表时间: 2021-07
• 期刊: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
• 作者: Wen-Chin Huang;Tomoki Hayashi;Xinjian Li;Shinji Watanabe;T. Toda

S3PRL-VC: Open-Source Voice Conversion Framework with Self-Supervised Speech Representations

• DOI: 10.1109/icassp43922.2022.9746430
• 发表时间: 2021-10
• 期刊: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
• 作者: Wen-Chin Huang;Shu-Wen Yang;Tomoki Hayashi;Hung-yi Lee;Shinji Watanabe;T. Toda