Highly expressive voices for machine video content localisation

用于机器视频内容本地化的高表现力声音

• 批准号: 73674
• 金额: $ 43.61万
• 依托单位: PAPERCUP TECHNOLOGIES LIMITED
• 依托单位国家: 英国
• 项目类别: Study
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=73674
• 关键词: Highly; expressive; voices; machine; video

Imagine any video available in any language, with both the unique qualities of the original actors' voices, and the unique way in which they delivered their lines, preserved in the new language. This is the ambitious vision that Papercup will make reality by harnessing the latest developments in the world of machine learning.In 2016, Google's Deepmind created the WaveNet vocoder. This was a revolution in speech synthesis. Prior to this, speech synthesis models were either concatenative (meaning that they work by glueing together short audio samples of recorded speech) or modelled methods, which generate speech "from scratch" using a model of how the human speech production system works. Concatenative synthesis typically resulted in more natural sounding voices, but with unnatural flow because the audio samples come from unrelated sections of speech. Modelled methods tended to produce better flow, but the voices sounded robotic. WaveNet is a deep-learning method, trained directly on audio samples, and combines the natural variation of modelled methods with the natural sound of concatenative methods. This development means that speech synthesis could become essentially indistinguishable from human speech.A vocoder (such as WaveNet), however, is not even half the story. You still have to tell it what to say, and how to say it. For a computer to achieve this, we must first recognise what was said in the original video, by whom, and in what way. Papercup exploits the latest developments in deep learning and has developed a patent-pending method for analysing the unique acoustic features of each speaker, and the way in which they delivered their lines. This is encoded by our algorithms using an internal learned representation, which enables the stresses, intonation, and emotion to be transferred across languages, in a manner analogous to the way translation tools translate text from one language to another.In this way, Papercup's approach replicates the unique vocal characteristics of the actors, and replicates their delivery. This has the potential to revolutionise the voiceover translation industry by creating faithful voiceover translations that accurately convey the original content in additional languages, and do so at scale with significantly lower costs than using traditional voiceover translation services with voice-actors.

想象一下，在任何一种语言的视频中，既有原声演员的独特品质，也有他们表达台词的独特方式，都被保留在新的语言中。Papercup将利用机器学习领域的最新发展，将这一雄心勃勃的愿景变为现实。2016年，b谷歌的Deepmind创建了WaveNet声码器。这是语音合成的一次革命。在此之前，语音合成模型要么是串联的（意思是它们通过将录制语音的短音频样本粘合在一起），要么是建模的方法，使用人类语音产生系统如何工作的模型“从零开始”生成语音。连接合成通常会产生更自然的声音，但不自然的流，因为音频样本来自不相关的语音部分。模拟的方法往往会产生更好的心流，但声音听起来像机器人。WaveNet是一种深度学习方法，直接在音频样本上进行训练，并将建模方法的自然变化与连接方法的自然声音相结合。这一发展意味着语音合成可以从本质上与人类语言无法区分。然而，声码器（如WaveNet）还不是故事的一半。你还得告诉它该说什么，怎么说。要让计算机实现这一点，我们必须首先识别原始视频中说了什么，是谁说的，以什么方式说的。Papercup利用了深度学习的最新发展，并开发了一种正在申请专利的方法，用于分析每个扬声器的独特声学特征，以及他们传递台词的方式。这是由我们的算法使用内部学习表示进行编码的，这使得重音，语调和情感能够跨语言传递，类似于翻译工具将文本从一种语言翻译成另一种语言的方式。通过这种方式，Papercup的方法复制了演员独特的声音特征，并复制了他们的表演。这有可能通过创建忠实的画外音翻译，以其他语言准确传达原始内容，从而彻底改变画外音翻译行业，并且与使用带有配音演员的传统画外音翻译服务相比，这样做的成本要低得多。