Scene Processing with Machine Learnable and Semantically Parametrized Scene Representations

点击翻译按钮获取中文摘要

Digital representations of visual reality and imagination is an integral part of almost all scientific disciplines, industry, and society. Imaging techniques and computer graphics have successfully solved the problems of creating accurate projections, e.g. image and video, of the real world over the last decades. However, such projections are fundamentally limited representations of the underlying scenes and only allow for passive consumption such as viewing on a screen. Many applications from image editing to augmented/virtual reality instead require active 3D exploration, creation, and editing of scenes, for which we need full virtual scene models. Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling. The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.