Autonomous Intelligent Machines & Systems

研究背景的简要描述，包括潜在的影响最近的生成模型[1，2，3]已经成功地生成了遵循用户提示的高质量视觉内容。然而，提示可能是模糊的，有必要提高生成的可控性，以便它可以用于现实生活中的应用。真实的内容可以提供细粒度的指导，这已经被用于控制对象的运动与视频运动传输[4，5]。可控架构，包括playablevideo工作线[6，7，8]，也可以通过对主体应用动作来交互式更新场景。改进这些方法将能够以多种形式（视频、3D、音频等）提供高质量的受控内容。拟议的研究可能会影响虚拟场景的创作过程，可能降低图形艺术家生成机器人模拟和视觉内容的成本。目的和目标的目的是开发新的生成方法来控制视觉场景，包括改变姿势，形状和外观或与环境交互的2D/3D对象。这利用了大规模模型的先验知识，通过文本、图像、草图、音频、视频或“学习”动作在可玩的环境中生成定制的内容。研究方法的新奇控制生成模型一直是一个长期存在的挑战，因为它旨在使内容在实践中可用。许多技术已经开发出来，以指导生成[9，10]，然而，他们缺乏对多种模态和细粒度控制的理解。通过微调进行的额外训练将其适用性限制在几代人的狭窄领域。现有的运动传输工作也难以从真实的内容中提取出真实的运动信息。理想情况下，预训练模型的全部功能可以在受控的同时保留。与EPSRC的战略和研究领域保持一致（该项目涉及EPSRC的研究领域）有关该领域的更多信息可以在www.example.com项目上找到http://www.epsrc.ac.uk/research/ourportfolio/researchareas/This涉及人工智能技术的研究领域（https：www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/artificial intelligence-technologies/），专门针对提高计算机视觉模型作为人类工具的自主和创造能力。任何参与的公司或合作者由Snap学生和合作伙伴资助，由Fabio Pizzati（牛津大学和MBZUAI）和Aliaksandr Siarohin（Snap）监督。

Brief description of the context of the research including potentialimpact Recent generative models [1, 2, 3] have been successful in generatinghigh quality visual content that follows a user prompt. However, prompts canbe ambiguous and it is necessary to improve the controllability of generationso that it can be used for real-life applications. Real content can provide fine grained guidance, which has been used in controlling the motion of objects withvideo motion transfer [4, 5]. Controllable architectures including the playablevideo line of work [6, 7, 8] also enables interactively updating a scene by applying actions to subjects. Improving these methods would enable high qualitycontrolled content in multiple modalities (video, 3D, audio etc.). The proposedresearch could impact the creative process of virtual scenes, potentially loweringcosts for generating robotic simulations and visual content by graphics artists.Aims and Objectives The aim is to develop novel generative methods tocontrol visual scenes including 2D/3D objects that change pose, shape andappearance or interact with the environment. This takes advantage of priorslearned by large-scale models to generate tailored content prompted throughtext, images, sketches, audio, video or 'learned' actions in a playable setting.Novelty of the research methodology Controlling generative models hasbeen a long-standing challenge as it aims to render the content usable in practice.Many techniques have been developed to guide generations [9, 10], however theylack understanding of multiple modalities and fine-grained control. Additionaltraining through finetuning limits their applicability to a narrow domain of generations. Existing motion transfer work also have difficulty extracting faithfuldisentangled motion information from real content. Ideally, the full capabilityof pre-trained models can be preserved while being controlled.Alignment to EPSRC's strategies and research areas (which EPSRCresearch area the project relates to) Further information on the areascan be found on http://www.epsrc.ac.uk/research/ourportfolio/researchareas/This project relates to the research area of Artificial Intelligence technologies(https://www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/artificial intelligence-technologies/), specifically targeting improving the autonomous andcreative abilities of computer vision models as a tool for humans.Any companies or collaborators involved Funded by Snap studentshipand co-supervised by Fabio Pizzati (University of Oxford and MBZUAI) andAliaksandr Siarohin (Snap).