权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Autonomous System for Sound Integration and GeneratioN (ASSIGN)

声音集成和生成自主系统 (ASSIGN)

基本信息

批准号：
EP/R005435/1
负责人：
Joshua Reiss
金额：
$ 14.84万
依托单位：
Queen Mary University of London
依托单位国家：
英国
项目类别：
Research Grant
财政年份：
2017
资助国家：
英国
起止时间：
2017 至无数据
项目状态：
已结题

来源：
https://gtr.ukri.org/projects?ref=EP%2FR005435%2F1
关键词：
Autonomous System Sound Integration GeneratioN

项目摘要

In sound design for immersive media and games, the designer typically starts by searching a large sound effects library for the desired sounds. The project team have delivered a highly successful InnovateUK feasibility project demonstrating that existing sound effects libraries can be replaced by sound synthesis techniques that use software algorithms to generate the desired sounds, enabling high quality sound effects across all forms of content creation. But the biggest challenge for professional sound design is the effort required to integrate sound effects into the timeline and story, and synchronise them with video or animation content. The Autonomous Systems for Sound Integration and GeneratioN (ASSIGN) project will deliver and validate a prototype sensor-driven sound synthesis service, capable of autonomous decision-making for use by anyone wanting to enhance or interact with sounds. It will allow synthesis of any sound, with integration into existing workflows,. The ASSIGN system generates sounds with their context from other sensor data. It uses animation storyboarding and visual object recognition information to automatically synthesize sounds and effects with their correct to drive sound generation, placement and perspective, thus enabling new forms of interaction. By exploiting sensors to generate sounds and their context, we give intelligence to the sound effects generation. This fits nicely with computer graphics approaches, where much of the animation is driven by some high level information, e.g., if a man drops a glass, we see it falling in the virtual world of the game, film or augmented reality. The animation is a property of the object, and sound effects should follow this same paradigm, thus enabling synchronisation of sound effects with CGI in immersive, game, film and augmented reality design.ASSIGN has the potential to revolutionize the sound design process for production of film, TV and animation content. By integrating such technologies with intuitive cloud media production tools developed by the lead partner, RPPtv, we can deliver a novel, sensor-driven SFX synthesis service. It democratises the industry, giving anyone the ability to become a sound designer, giving users control, harnessing their creativity. Initial work by the academic partner in this area has shown great promise. A wide range of sounds, including sound textures, soundscapes and impulse sounds covering most popular sound effects can be synthesized in real-time with high quality and relevant user controls. But significant research questions remain unanswered; - How rich and relevant are available metadata for control and placement of synthesized sound effects?- How effective are state-of-the-art object and scene recognition and object tracking methods for extracting details of an acoustic environment?- How versatile are the sound synthesis models?- How can the parameters of sound synthesis models be mapped to intuitive controls?- How should autonomous sound effect generation and integration methods be evaluated? The project is industry-led, with an experienced academic partner and validated with industry users. It exploits the outputs of a highly successful InnovateUK Feasibility Study and groundbreaking research from Queen Mary University of London's audio engineering team. It brings together knowledge, skills and technologies from immersive and games audio production, cloud service delivery and academic research excellence. A demonstrator of the service will be evaluated and validated in audio post-production with immersive, object-based and 3D audio specialists Gareth Llewelyn Sound. The business potential is compelling, since the project will demonstrate a disruptive cloud service for a globally used and purchased media resource. The outputs will include a prototype for CGI-linked SFX production, with full market analysis, business models and road map to launch a commercial service.

在沉浸式媒体和游戏的声音设计中，设计师通常从搜索大型音效库开始寻找所需的声音。该项目团队已经交付了一个非常成功的InnovateUK可行性项目，证明现有的声音效果库可以被声音合成技术所取代，该技术使用软件算法来生成所需的声音，从而在所有形式的内容创作中实现高质量的声音效果。但专业音效设计面临的最大挑战是将音效整合到时间轴和故事中，并与视频或动画内容同步。声音集成和生成自主系统（ASSIGN）项目将提供并验证原型传感器驱动的声音合成服务，能够自主决策，供任何想要增强声音或与声音互动的人使用。它将允许合成任何声音，并集成到现有的工作流程中。ASSIGN系统根据其他传感器数据生成具有上下文的声音。它使用动画故事板和视觉对象识别信息自动合成声音和效果，并正确驱动声音生成，放置和视角，从而实现新的交互形式。通过利用传感器来生成声音及其上下文，我们为音效生成提供了智能。这非常适合计算机图形方法，其中大部分动画是由一些高级信息驱动的，例如，如果一个人掉了一个杯子，我们会在游戏、电影或增强现实的虚拟世界中看到它掉下来。动画是物体的属性，音效也应该遵循同样的模式，从而在沉浸式、游戏、电影和增强现实设计中实现音效与CGI的同步。ASSIGN有可能彻底改变电影、电视和动画内容制作的声音设计过程。通过将这些技术与领先合作伙伴RPPTv开发的直观云媒体制作工具相结合，我们可以提供一种新颖的传感器驱动的SFX合成服务。它使这个行业民主化，让任何人都有能力成为一个声音设计师，让用户控制，利用他们的创造力。学术伙伴在这一领域的初步工作显示出很大的希望。通过高质量和相关的用户控制，可以实时合成各种声音，包括声音纹理、音景和脉冲声音，涵盖最流行的音效。但重要的研究问题仍然没有答案; -如何丰富和相关的可用元数据的控制和放置合成的声音效果？最先进的物体和场景识别以及物体跟踪方法在提取声学环境细节方面的效果如何？声音合成模型的用途有多广？如何将声音合成模型的参数映射到直观的控制？如何评估自主音效生成和集成方法？该项目由行业主导，拥有经验丰富的学术合作伙伴，并经过行业用户的验证。它利用了非常成功的InnovateUK可行性研究的成果和伦敦大学玛丽女王音频工程团队的开创性研究。它汇集了沉浸式和游戏音频制作，云服务交付和学术研究卓越的知识，技能和技术。该服务的演示者将在音频后期制作中与沉浸式，基于对象和3D音频专家Gareth Llewelyn Sound进行评估和验证。商业潜力是引人注目的，因为该项目将为全球使用和购买的媒体资源展示颠覆性的云服务。产出将包括与CGI相关的SFX生产原型，以及全面的市场分析、商业模式和启动商业服务的路线图。