Autonomous Intelligent Machines & Systems

最近的生成模型[1,2,3]已经成功地根据用户提示生成高质量的视觉内容。然而，提示可能是模糊的，有必要提高生成的可控性，以便它可以用于现实生活中的应用程序。真实的内容可以提供细粒度的引导，这已经被用于视频运动传输中控制物体的运动[4,5]。可控架构包括playablevideo工作线[6,7,8]也可以通过对主体应用动作来交互式地更新场景。改进这些方法可以实现多种形式（视频、3D、音频等）的高质量受控内容。这项提议的研究可能会影响虚拟场景的创作过程，潜在地降低图形艺术家生成机器人模拟和视觉内容的成本。目的是开发新的生成方法来控制视觉场景，包括改变姿势，形状和外观或与环境交互的2D/3D对象。这利用了大规模模型学习到的先验知识，在可玩的环境中通过文本、图像、草图、音频、视频或“学习”动作来生成定制内容。研究方法的新颖性控制生成模型一直是一个长期的挑战，因为它旨在使内容在实践中可用。已经开发了许多技术来指导代[9,10]，但是它们缺乏对多模式和细粒度控制的理解。通过微调进行的额外训练将它们的适用性限制在一个狭窄的世代范围内。现有的运动传输工作也难以从真实内容中提取出真实的运动信息。理想情况下，预训练模型的全部能力可以在被控制的同时被保留。与EPSRC的战略和研究领域（项目所涉及的EPSRC研究领域）保持一致，有关该领域的进一步信息可在http://www.epsrc.ac.uk/research/ourportfolio/researchareas/This上找到，该项目涉及人工智能技术的研究领域（https://www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/artificial Intelligence -technologies/）。具体目标是提高计算机视觉模型作为人类工具的自主和创造能力。由Snap学生资助，Fabio Pizzati（牛津大学和MBZUAI）和aliaksandr Siarohin （Snap）共同监督的任何公司或合作者。

Brief description of the context of the research including potentialimpact Recent generative models [1, 2, 3] have been successful in generatinghigh quality visual content that follows a user prompt. However, prompts canbe ambiguous and it is necessary to improve the controllability of generationso that it can be used for real-life applications. Real content can provide fine grained guidance, which has been used in controlling the motion of objects withvideo motion transfer [4, 5]. Controllable architectures including the playablevideo line of work [6, 7, 8] also enables interactively updating a scene by applying actions to subjects. Improving these methods would enable high qualitycontrolled content in multiple modalities (video, 3D, audio etc.). The proposedresearch could impact the creative process of virtual scenes, potentially loweringcosts for generating robotic simulations and visual content by graphics artists.Aims and Objectives The aim is to develop novel generative methods tocontrol visual scenes including 2D/3D objects that change pose, shape andappearance or interact with the environment. This takes advantage of priorslearned by large-scale models to generate tailored content prompted throughtext, images, sketches, audio, video or 'learned' actions in a playable setting.Novelty of the research methodology Controlling generative models hasbeen a long-standing challenge as it aims to render the content usable in practice.Many techniques have been developed to guide generations [9, 10], however theylack understanding of multiple modalities and fine-grained control. Additionaltraining through finetuning limits their applicability to a narrow domain of generations. Existing motion transfer work also have difficulty extracting faithfuldisentangled motion information from real content. Ideally, the full capabilityof pre-trained models can be preserved while being controlled.Alignment to EPSRC's strategies and research areas (which EPSRCresearch area the project relates to) Further information on the areascan be found on http://www.epsrc.ac.uk/research/ourportfolio/researchareas/This project relates to the research area of Artificial Intelligence technologies(https://www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/artificial intelligence-technologies/), specifically targeting improving the autonomous andcreative abilities of computer vision models as a tool for humans.Any companies or collaborators involved Funded by Snap studentshipand co-supervised by Fabio Pizzati (University of Oxford and MBZUAI) andAliaksandr Siarohin (Snap).