Augmented speech communication using multi-modal signals with real-time, low-latency voice conversion

本研究的目的是在多模态信号和实时处理技术的帮助下，应用语音转换（VC）来实现现实世界应用的交互式语音生产范式。在第二年，申请人着重于三个方面。(1)持续改进基础VC技术，特别是基于自监督语音表示（S3R）的VC，这是一种降低训练数据需求的新兴趋势。申请人持续更新用于研究人员评估VC的S3R模型的开源工具包S3PRL-VC，并在IEEE Journal of Selected Topics in Signal Processing上发表最新实验结果。(2)外国口音转换，减少外国口音，提高沟通效率。向一次国际会议提交了一份文件，该文件对目前的方法进行了统一评价，并确定了尚未解决的问题，目前正在审查中。(3)歌唱声音转换是一种具有增强人类交流能力潜力的基本技术。申请人正在举办名为“歌唱声音转换挑战2023”的科学活动，旨在提供包括任务和数据集在内的统一实验环境，以吸引世界各地的研究人员研究这一问题，并探索最先进技术的局限性。

The purpose of this research is to apply voice conversion (VC) to realize an interactive speech production paradigm for real-world applications, with the help of multimodal signals and real-time processing techniques. In the second year, the applicant focused on three aspects.(1) Continued improvement on fundamental VC techniques, specifically self-supervised speech representation (S3R)-based VC, an emerging trend which reduces training data requirements. The applicant kept on updating S3PRL-VC, an open-source toolkit for researchers to evaluate S3R models for VC, and published the latest experimental results in the IEEE Journal of Selected Topics in Signal Processing.(2) Foreign accent conversion, a task that helps reduce foreign accents for efficient communication. A paper that provides an unified evaluation of current approaches and identifies unsolved problems is submitted to an international conference and currently under review.(3) Singing voice conversion, a fundamental technique that has the potential to augment the communication ability of human. The applicant is running a scientific event named the Singing Voice Conversion Challenge 2023, which aims to provide an unified experimental setting including task and dataset, in order to attract researchers world-wide to look into this problem and explore the limitation of the state-of-the-art techniques.