ReBabel - deep learning cross-lingual vocal characteristic matching with automated lip-sync

在21亿英镑/年（2019年）和5.9%的复合年增长率（Marketwatch.com）下，对外语制作的需求不断增长，这是由品牌全球化，流媒体服务（Netflix/亚马逊Prime），社交平台（YouTube/Twitch）和智能手机的普及所推动的。欧洲约占全球配音行业的22%。然而，尽管观众数据和调查都显示配音内容比字幕更受欢迎，但跨语言配音的使用受到成本高和制作时间长的限制。其他挑战包括：* 改编脚本的难度和需要多次拍摄，以便配音演员（VA）可以匹配对口型。糟糕的对口型很不吸引人，有时还很滑稽。铸造理想情况下，寻求与原始参与者的密切匹配，但受到时间和成本的限制，甚至可能导致“最适合”的方法和/或重复使用VA。声音/身体不匹配导致的美学不连贯。当与多位明星相关的VA的可用性有限时，时间瓶颈。因此，制片人必须平衡配音的额外成本和增加销售的潜力，而不是只提供字幕。**项目愿景 ** 在此项目中，我们将利用语音合成/深度学习（DL）专业知识，创建ReBabel，这是一种创新且独特的跨语言配音工具，可通过关键创新降低成本，提高质量并应对挑战：1.跨语言变声：改变外语表演，使其听起来像是原表演者的声音。我们独特的技术通过利用我们独特的神经网络模型将语音转换为概念并返回，从而实现语音到语音合成（将一种语音转换为另一种不同语言的语音）。我们的概念抽象既传达了文字的文本意义/信息，也传达了可以修改意义、赋予细微差别和传达情感的非语言学（有时称为声乐）信息，并且可以进行编辑以应用所需的转换。Audio-LipSync：人工智能驱动的手动/自动音频重新同步，以匹配视频中的嘴唇运动。这将涉及直接修改音频通道以匹配演员的嘴唇，同时传达预期的声音信息/内容/表演。ReBabel将在我们正在开发的Voice Studio套件中形成一个模块，最初将在最终的消费者/专业消费者版本之前向专业制作公司提供“Photoshop for Voice”。**关键目标 * 开发快速、准确和高质量的机器学习（ML）模型和支持数据结构。针对主要国际语言实施DL，以实现语音/视频口型同步。*模型结果的测试和验证。实现用户技术知识（可用性）的低标准。

At £2.1bn/annum (2019) and 5.9% CAGR (Marketwatch.com) there is growing demand for foreign-language production, driven by brand globalisation, streaming services (Netflix/Amazon Prime), social platforms (YouTube/Twitch) and the ubiquity of smart phones. Europe accounts for ~22% of the global dubbing industry.However, while audience data and surveys both show a preference for dubbed content over subtitles, the use of cross-lingual dubbing is limited by high costs and long production times. Other challenges include:* The difficulty of adapting scripts and the need for multiple takes so Voice Actors (VAs) can match lip-synch. Poor lip-synch is unappealing and sometimes comical.* Casting. Ideally a close match to the original actor is sought but is constrained by time and costs that may even result in a "best-fit" approach and/or reusing VAs.* Aesthetic incoherence arising from a voice/body mismatch.* Time bottlenecks when VAs associated with multiple stars have limited availability.Producers must therefore balance the additional cost of a dub and the potential for increased sales against offering subtitles only.**Vision for the project**In this project we will build on our Speech-Synthesis/Deep Learning (DL) expertise to create ReBabel, an innovative and unique new cross-lingual dubbing tool that lowers costs, improves quality and addresses the challenges via key innovations:1. Cross-lingual voice-morphing: Changing a foreign-language performance to sound like it was given by the original performer. Our unique technology enables speech-to-speech synthesis (transforming one voice to another in a different language) by utilising our unique neural network model that transforms speech into concepts and back. Our conceptual abstractions convey both literal textual meaning/information, as well as the paralinguistics (sometimes called vocalics) information that can modify meaning, give nuance, and convey emotion and can be edited to apply the desired transformations.2. Audio-LipSync: AI-driven manual/automatic re-synchronising of the audio to match the lip movements in the video. This will involve direct modification of the audio channel to match the lips of the actor, while conveying the vocal message/content/performance that is expected.ReBabel will form a module within our in-development Voice Studio suite that will initially offer "Photoshop for Voice" to professional production companies before eventual consumer/prosumer versions.**Key objectives*** Development of a fast, accurate and high-quality Machine Learning (ML) model and supporting data structures.* Implementation of DL for key international languages to enable voice/video lip-sync.* Testing and validation of model results.* Achieving a low bar for user's technical knowledge (usability).