ReBabel - deep learning cross-lingual vocal characteristic matching with automated lip-sync

在品牌全球化、流媒体服务（Netflix/Amazon Prime）、社交平台（YouTube/Twitch）和无处不在的智能手机的推动下，对外语制作的需求不断增长，每年（2019年）为21亿英镑，复合年增长率为5.9% （Marketwatch.com）。欧洲占全球配音产业的约22%。然而，虽然观众数据和调查都显示，配音内容比字幕更受欢迎，但跨语言配音的使用受到成本高和制作时间长的限制。其他挑战包括：*改编剧本的困难，需要多次拍摄，以便配音演员（VAs）能够匹配假唱。糟糕的假唱没有吸引力，有时还很滑稽。*铸造。理想情况下，会寻求与原始参与者的密切匹配，但受时间和成本的限制，甚至可能导致“最适合”的方法和/或重用VAs。*由声音/身体不匹配引起的美学不连贯。*当与多星关联的VAs可用性有限时，时间瓶颈。因此，制片人必须权衡配音的额外成本和增加销售的潜力，而不是只提供字幕。在这个项目中，我们将建立在我们的语音合成/深度学习（DL）专业知识的基础上，创建ReBabel，这是一种创新和独特的新型跨语言配音工具，可以降低成本，提高质量，并通过关键创新解决挑战：跨语言变声：把外语表演改成原声表演者的声音。我们独特的技术通过利用我们独特的神经网络模型，将语音转换为概念，从而实现语音到语音的合成（将一种声音转换为另一种不同语言的声音）。我们的概念抽象既传达了文本的字面意义/信息，也传达了副语言学（有时称为语音）信息，这些信息可以修改意义，提供细微差别，传达情感，并可以进行编辑以应用所需的转换。audio - lipsync：人工智能驱动的手动/自动重新同步音频，以匹配视频中的嘴唇运动。这将涉及直接修改音频通道以匹配演员的嘴唇，同时传达预期的声音信息/内容/表演。ReBabel将在我们正在开发的Voice Studio套件中形成一个模块，在最终的消费者/专业消费者版本之前，最初将为专业制作公司提供“Photoshop for Voice”。**主要目标***开发快速，准确和高质量的机器学习（ML）模型和支持数据结构。*为主要国际语言实施深度学习，以实现语音/视频对口型。*测试和验证模型结果。*实现用户技术知识（可用性）的低门槛。

At £2.1bn/annum (2019) and 5.9% CAGR (Marketwatch.com) there is growing demand for foreign-language production, driven by brand globalisation, streaming services (Netflix/Amazon Prime), social platforms (YouTube/Twitch) and the ubiquity of smart phones. Europe accounts for ~22% of the global dubbing industry.However, while audience data and surveys both show a preference for dubbed content over subtitles, the use of cross-lingual dubbing is limited by high costs and long production times. Other challenges include:* The difficulty of adapting scripts and the need for multiple takes so Voice Actors (VAs) can match lip-synch. Poor lip-synch is unappealing and sometimes comical.* Casting. Ideally a close match to the original actor is sought but is constrained by time and costs that may even result in a "best-fit" approach and/or reusing VAs.* Aesthetic incoherence arising from a voice/body mismatch.* Time bottlenecks when VAs associated with multiple stars have limited availability.Producers must therefore balance the additional cost of a dub and the potential for increased sales against offering subtitles only.**Vision for the project**In this project we will build on our Speech-Synthesis/Deep Learning (DL) expertise to create ReBabel, an innovative and unique new cross-lingual dubbing tool that lowers costs, improves quality and addresses the challenges via key innovations:1. Cross-lingual voice-morphing: Changing a foreign-language performance to sound like it was given by the original performer. Our unique technology enables speech-to-speech synthesis (transforming one voice to another in a different language) by utilising our unique neural network model that transforms speech into concepts and back. Our conceptual abstractions convey both literal textual meaning/information, as well as the paralinguistics (sometimes called vocalics) information that can modify meaning, give nuance, and convey emotion and can be edited to apply the desired transformations.2. Audio-LipSync: AI-driven manual/automatic re-synchronising of the audio to match the lip movements in the video. This will involve direct modification of the audio channel to match the lips of the actor, while conveying the vocal message/content/performance that is expected.ReBabel will form a module within our in-development Voice Studio suite that will initially offer "Photoshop for Voice" to professional production companies before eventual consumer/prosumer versions.**Key objectives*** Development of a fast, accurate and high-quality Machine Learning (ML) model and supporting data structures.* Implementation of DL for key international languages to enable voice/video lip-sync.* Testing and validation of model results.* Achieving a low bar for user's technical knowledge (usability).