ReBabel - deep learning cross-lingual vocal characteristic matching with automated lip-sync

在品牌全球化、流媒体服务(Netflix/Amazon Prime)、社交平台(YouTube/Twitch)和智能手机无处不在的推动下，外语制作的需求不断增长，分别为21亿GB/年(2019年)和5.9%的CAGR(Marketwatch.com)。欧洲约占全球配音业的22%。然而，尽管观众数据和调查都显示，相对于字幕，观众更喜欢配音内容，但跨语言配音的使用受到成本高和制作时间长的限制。其他挑战包括：*改编剧本的难度，以及需要多个镜头才能让配音演员(VAS)对口型进行匹配。糟糕的假唱不吸引人，有时还很滑稽。*选角。理想的情况是，寻找与原演员非常匹配的演员，但受时间和成本的限制，这甚至可能导致最佳匹配的方法和/或重复使用VAS。*声音/身体不匹配导致的审美不连贯。*与多个明星相关的VAS的时间瓶颈。因此，制片人必须权衡配音的额外成本和增加销售的可能性，而不是只提供字幕。**对项目的愿景**在这个项目中，我们将基于我们的语音合成/深度学习(DL)专业知识来创建ReBabel，这是一种创新且独特的跨语言配音工具，可以降低成本，通过关键创新提高质量并应对挑战：1.跨语言语音变形：将外语表演改变为听起来像是原始表演者给出的声音。我们独特的技术通过利用我们独特的神经网络模型实现语音到语音的合成(在不同的语言中将一种声音转换为另一种声音)，该模型将语音转换为概念并返回。我们的概念抽象既传达了字面上的文本意义/信息，也传达了副语言学(有时称为发音)信息，这些信息可以修改意义、赋予细微差别和传达情感，并可以编辑以应用所需的转换。音频-口型同步：人工智能驱动的手动/自动音频重新同步，以匹配视频中的嘴唇运动。这将包括直接修改音频通道以匹配演员的嘴唇，同时传达预期的声音信息/内容/表演。ReBabel将在我们正在开发的Voice Studio套件中形成一个模块，该模块最初将在最终的消费者/消费者版本之前向专业制作公司提供“Photoshop for Voice”。**主要目标**开发快速、准确和高质量的机器学习(ML)模型和支持数据结构。*为关键的国际语言实施DL，以实现语音/视频唇形同步。*测试和验证模型结果。*为用户的技术知识(可用性)实现较低的标准。

At £2.1bn/annum (2019) and 5.9% CAGR (Marketwatch.com) there is growing demand for foreign-language production, driven by brand globalisation, streaming services (Netflix/Amazon Prime), social platforms (YouTube/Twitch) and the ubiquity of smart phones. Europe accounts for ~22% of the global dubbing industry.However, while audience data and surveys both show a preference for dubbed content over subtitles, the use of cross-lingual dubbing is limited by high costs and long production times. Other challenges include:* The difficulty of adapting scripts and the need for multiple takes so Voice Actors (VAs) can match lip-synch. Poor lip-synch is unappealing and sometimes comical.* Casting. Ideally a close match to the original actor is sought but is constrained by time and costs that may even result in a "best-fit" approach and/or reusing VAs.* Aesthetic incoherence arising from a voice/body mismatch.* Time bottlenecks when VAs associated with multiple stars have limited availability.Producers must therefore balance the additional cost of a dub and the potential for increased sales against offering subtitles only.**Vision for the project**In this project we will build on our Speech-Synthesis/Deep Learning (DL) expertise to create ReBabel, an innovative and unique new cross-lingual dubbing tool that lowers costs, improves quality and addresses the challenges via key innovations:1. Cross-lingual voice-morphing: Changing a foreign-language performance to sound like it was given by the original performer. Our unique technology enables speech-to-speech synthesis (transforming one voice to another in a different language) by utilising our unique neural network model that transforms speech into concepts and back. Our conceptual abstractions convey both literal textual meaning/information, as well as the paralinguistics (sometimes called vocalics) information that can modify meaning, give nuance, and convey emotion and can be edited to apply the desired transformations.2. Audio-LipSync: AI-driven manual/automatic re-synchronising of the audio to match the lip movements in the video. This will involve direct modification of the audio channel to match the lips of the actor, while conveying the vocal message/content/performance that is expected.ReBabel will form a module within our in-development Voice Studio suite that will initially offer "Photoshop for Voice" to professional production companies before eventual consumer/prosumer versions.**Key objectives*** Development of a fast, accurate and high-quality Machine Learning (ML) model and supporting data structures.* Implementation of DL for key international languages to enable voice/video lip-sync.* Testing and validation of model results.* Achieving a low bar for user's technical knowledge (usability).