Exemplar-based Expressive Speech Synthesis

合成语音正变得无处不在：家里的“智能”扬声器，公共交通上的广播系统，以及电话线路上的语音助手。公众对能够被我们的笑话逗乐的“更聪明”的助手有着强烈的需求；作为鼓励和强调的导师与我们的孩子互动；打电话问候我们的父母；为孤立的人提供安慰的“倾听者”；并提供平静和支持性的虚拟治疗。为了支持当前和未来的应用，语音合成技术需要满足许多要求。首先，它需要为快速研究和开发而定制，其次，它需要能够产生任何口语内容，包括富有表现力的语音特征。然而，目前没有一种合成技术能够同时满足上述所有要求。例如，虽然目前的非机器学习方法允许将预先录制的短语有效地组合成完整的句子，但这也意味着必须先记录缺失的必要短语，从而限制了它们的灵活性和效率。另一方面，当前的机器学习模型可以无缝地合成任何口头内容。然而，创建这样的模型是一个非常昂贵、耗时和计算要求很高的过程。此外，这些模型对语音特征的质量控制非常有限，缺乏可解释性，这在研究和商业环境中都是非常理想的条件。在这个项目中，目标是通过借鉴认知科学中的“范例”概念，开发一种计算效率高、可定制、富有表现力和可解释的语音合成。在认知科学领域，“范例”和“原型”的概念构成了人类如何对概念进行分类的重要观点的一部分。特别是，范例理论认为，单一的例子，而不是原型（例子的平均），构成了我们如何理解世界和与世界互动的基本组成部分。支持范例理论的关键论点是我们作为人类解决基于几个例子的复杂任务的能力，这使得该理论对涉及复杂现象或需要高计算效率的应用具有吸引力。此外，表达性语音合成结合了表达性和语音产生，这是两个仍然知之甚少的复杂现象。与原型理论不同，范例理论，至少在理论上，能够产生表达性言语，只要至少有一个期望的口语内容的记录和一个具有期望的表达性的记录是可用的。最后，采用范例理论通过使用真实的例子来提高决策过程的透明度，这些例子可以在任务中进行检查、修改、替换、添加等。该目标将通过三种创新手段实现：i)为基于范例的语音合成制定方法框架，ii)从预先录制的示例中构建基于范例的语音表现力表示，以及iii)提出一种将这种基于表达性的表示整合到i)框架中的新方法。

Synthetic voices are becoming ubiquitous: `smart' speakers at home, announcement systems on public transport, and voice-enabled assistants on call lines. There exist a strong public demand for `smarter' assistants capable of laughing at our jokes; interacting with our children as encouraging and emphatic tutors; calling to check up on our parents; providing a reassuring `ear' for an isolated person; and offering calming and supportive virtual therapy. To support current and future applications, voice synthesis technology needs to satisfy a number of requirements. First, it needs to be customisable for rapid research and development, and second, it needs to be able to produce any spoken content, including expressive voice characteristics. However, none of the current synthesis technologies can simultaneously satisfy all of the above requirements. For instance, while current non-machine learning approaches allow pre-recorded phrases to be efficiently combined into complete sentences, it also means that missing necessary phrases must be recorded first, thereby limiting their flexibility and efficiency. On the other hand, current machine learning models can seamlessly synthesise any spoken content. However, creating such models is a very costly, time-consuming and computationally demanding process. Furthermore, these models offer a very limited control over the qualities of the voice characteristics and lack interpretability, which are highly desirable conditions in both research and commercial settings.In this project, the objective is to develop a computationally efficient, customisable, expressive and interpretable speech synthesis, by drawing from the concept of `exemplars' in cognitive science.In the field of cognitive science, the notions of `exemplars' and `prototypes' form a part of a prominent view on how humans categorise concepts. In particular, exemplar theory argues that singular examples, rather than prototypes (an average of examples), form the basic building blocks of how we understand and interact with the world. The key argument in favour of exemplar theory is our ability as humans to solve complex tasks based on just a few examples, which makes this theory appealing to applications that involve complex phenomena or that require high computational efficiency. Furthermore, expressive speech synthesis combines expressivity and speech production, which are two complex phenomena that remain poorly understood. Unlike prototype theory, exemplar theory, at least theoretically, enables to produce expressive speech, provided that at least one recording of the desired spoken content and one recording featuring the desired expressivity are available. Lastly, adopting exemplar theory promotes transparency during the decision making process through the use of real examples that can be inspected, modified, replaced, added, etc. within the task.The objective will be achieved through three innovative means by: i) formulating a methodological framework for exemplar-based speech synthesis, ii) building an exemplar-based representation for speech expressivity from pre-recorded examples and iii) presenting a novel methodology for integrating this expressivity-based representation into the framework of i).