Autonomous System for Sound Integration and GeneratioN (ASSIGN)

在沉浸式媒体和游戏的音效设计中，设计者通常从搜索大型音效库来寻找所需的声音开始。项目团队已经完成了一个非常成功的InnovateUK可行性项目，展示了现有的音效库可以被声音合成技术取代，这种技术使用软件算法来生成所需的声音，从而实现所有形式的内容创作的高质量声音效果。但专业音效设计面临的最大挑战是，需要努力将音效整合到时间轴和故事中，并将它们与视频或动画内容同步。声音集成和生成自主系统(ASSIGN)项目将提供和验证传感器驱动的声音合成服务原型，该服务能够自主决策，供任何想要增强声音或与声音互动的人使用。它将允许合成任何声音，并集成到现有的工作流中。分配系统根据其他传感器数据生成声音及其上下文。它使用动画故事板和视觉对象识别信息，以正确的方式自动合成声音和效果，以驱动声音的生成、放置和透视，从而实现新的交互形式。通过利用传感器来生成声音及其上下文，我们为音效生成提供了智能。这与计算机图形学方法非常吻合，在计算机图形学方法中，大部分动画都是由一些高级信息驱动的，例如，如果一个人掉了一个杯子，我们可以看到它掉在游戏、电影或增强现实的虚拟世界中。动画是物体的属性，音效应该遵循相同的范例，从而使沉浸式、游戏、电影和增强现实设计中的音效与CGI同步。ASSIGN有可能彻底改变电影、电视和动画内容制作的音效设计过程。通过将这些技术与主要合作伙伴RPPtv开发的直观云媒体制作工具相集成，我们可以提供新颖的、传感器驱动的SFX合成服务。它让这个行业大众化，让任何人都有能力成为声音设计师，让用户控制，利用他们的创造力。这位学术伙伴在这一领域的初步工作显示出了巨大的希望。涵盖大多数流行音效的各种声音，包括声音纹理、音景和脉冲声音，都可以通过高质量和相关的用户控制进行实时合成。但重要的研究问题仍未得到解答；-可用于控制和放置合成音效的元数据有多丰富和相关？-用于提取声环境细节的最先进的对象和场景识别和对象跟踪方法有多有效？-声音合成模型的通用性有多大？-声音合成模型的参数如何映射到直观的控制？-应该如何评估自主的音效生成和集成方法？该项目是行业领先的，拥有经验丰富的学术合作伙伴，并得到行业用户的验证。它利用了一项非常成功的InnovateUK可行性研究和伦敦玛丽女王大学音频工程团队的开创性研究的成果。它汇集了沉浸式和游戏音频制作、云服务交付和卓越学术研究的知识、技能和技术。这项服务的演示者将在音频后期制作中与身临其境的、基于对象的和3D音频专家Gareth Llewell yn Sound一起进行评估和验证。该项目的商业潜力是引人注目的，因为该项目将展示一种面向全球使用和购买的媒体资源的颠覆性云服务。这些成果将包括与CGI挂钩的SFX生产的原型，以及全面的市场分析、商业模式和推出商业服务的路线图。

In sound design for immersive media and games, the designer typically starts by searching a large sound effects library for the desired sounds. The project team have delivered a highly successful InnovateUK feasibility project demonstrating that existing sound effects libraries can be replaced by sound synthesis techniques that use software algorithms to generate the desired sounds, enabling high quality sound effects across all forms of content creation. But the biggest challenge for professional sound design is the effort required to integrate sound effects into the timeline and story, and synchronise them with video or animation content. The Autonomous Systems for Sound Integration and GeneratioN (ASSIGN) project will deliver and validate a prototype sensor-driven sound synthesis service, capable of autonomous decision-making for use by anyone wanting to enhance or interact with sounds. It will allow synthesis of any sound, with integration into existing workflows,. The ASSIGN system generates sounds with their context from other sensor data. It uses animation storyboarding and visual object recognition information to automatically synthesize sounds and effects with their correct to drive sound generation, placement and perspective, thus enabling new forms of interaction. By exploiting sensors to generate sounds and their context, we give intelligence to the sound effects generation. This fits nicely with computer graphics approaches, where much of the animation is driven by some high level information, e.g., if a man drops a glass, we see it falling in the virtual world of the game, film or augmented reality. The animation is a property of the object, and sound effects should follow this same paradigm, thus enabling synchronisation of sound effects with CGI in immersive, game, film and augmented reality design.ASSIGN has the potential to revolutionize the sound design process for production of film, TV and animation content. By integrating such technologies with intuitive cloud media production tools developed by the lead partner, RPPtv, we can deliver a novel, sensor-driven SFX synthesis service. It democratises the industry, giving anyone the ability to become a sound designer, giving users control, harnessing their creativity. Initial work by the academic partner in this area has shown great promise. A wide range of sounds, including sound textures, soundscapes and impulse sounds covering most popular sound effects can be synthesized in real-time with high quality and relevant user controls. But significant research questions remain unanswered; - How rich and relevant are available metadata for control and placement of synthesized sound effects?- How effective are state-of-the-art object and scene recognition and object tracking methods for extracting details of an acoustic environment?- How versatile are the sound synthesis models?- How can the parameters of sound synthesis models be mapped to intuitive controls?- How should autonomous sound effect generation and integration methods be evaluated? The project is industry-led, with an experienced academic partner and validated with industry users. It exploits the outputs of a highly successful InnovateUK Feasibility Study and groundbreaking research from Queen Mary University of London's audio engineering team. It brings together knowledge, skills and technologies from immersive and games audio production, cloud service delivery and academic research excellence. A demonstrator of the service will be evaluated and validated in audio post-production with immersive, object-based and 3D audio specialists Gareth Llewelyn Sound. The business potential is compelling, since the project will demonstrate a disruptive cloud service for a globally used and purchased media resource. The outputs will include a prototype for CGI-linked SFX production, with full market analysis, business models and road map to launch a commercial service.