Autonomous Intelligent Machines & Systems

点击翻译按钮获取中文摘要

Brief description of the context of the research including potentialimpact Recent generative models [1, 2, 3] have been successful in generatinghigh quality visual content that follows a user prompt. However, prompts canbe ambiguous and it is necessary to improve the controllability of generationso that it can be used for real-life applications. Real content can provide fine grained guidance, which has been used in controlling the motion of objects withvideo motion transfer [4, 5]. Controllable architectures including the playablevideo line of work [6, 7, 8] also enables interactively updating a scene by applying actions to subjects. Improving these methods would enable high qualitycontrolled content in multiple modalities (video, 3D, audio etc.). The proposedresearch could impact the creative process of virtual scenes, potentially loweringcosts for generating robotic simulations and visual content by graphics artists.Aims and Objectives The aim is to develop novel generative methods tocontrol visual scenes including 2D/3D objects that change pose, shape andappearance or interact with the environment. This takes advantage of priorslearned by large-scale models to generate tailored content prompted throughtext, images, sketches, audio, video or 'learned' actions in a playable setting.Novelty of the research methodology Controlling generative models hasbeen a long-standing challenge as it aims to render the content usable in practice.Many techniques have been developed to guide generations [9, 10], however theylack understanding of multiple modalities and fine-grained control. Additionaltraining through finetuning limits their applicability to a narrow domain of generations. Existing motion transfer work also have difficulty extracting faithfuldisentangled motion information from real content. Ideally, the full capabilityof pre-trained models can be preserved while being controlled.Alignment to EPSRC's strategies and research areas (which EPSRCresearch area the project relates to) Further information on the areascan be found on http://www.epsrc.ac.uk/research/ourportfolio/researchareas/This project relates to the research area of Artificial Intelligence technologies(https://www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/artificial intelligence-technologies/), specifically targeting improving the autonomous andcreative abilities of computer vision models as a tool for humans.Any companies or collaborators involved Funded by Snap studentshipand co-supervised by Fabio Pizzati (University of Oxford and MBZUAI) andAliaksandr Siarohin (Snap).