Scene Processing with Machine Learnable and Semantically Parametrized Scene Representations

视觉现实和想象力的数字表示几乎是所有科学学科，工业和社会的组成部分。在过去的几十年里，成像技术和计算机图形学已经成功地解决了创建真实的世界的精确投影（例如图像和视频）的问题。然而，这样的投影基本上是底层场景的有限表示，并且仅允许被动消费，诸如在屏幕上观看。从图像编辑到增强/虚拟现实的许多应用都需要主动的3D探索，创建和编辑场景，我们需要完整的虚拟场景模型。通过数字建模或捕获创建作为真实的世界及其他世界的高保真模型的虚拟场景，传统上只有受过教育的艺术家和工程师才能使用复杂的软件和硬件工具工作无数个小时。这些专业人员制作和使用精心设计的几何，外观和运动参数化，这使他们能够创作和编辑虚拟场景与现实和他们的想象力的所有复杂细节。另一方面，许多计算机视觉技术试图通过在不受控制的环境中经由更简单的传感器从真实的世界捕获场景，或者使用仅隐式编码场景语义的算法（例如，在人工神经网络的潜在空间中）来改变场景的图像，从而具有捷径。这些导致较低质量的场景表示，并且形式不容易编辑以进行语义有意义的建模。该研究项目的目标是解决这些缺点，并开发一种场景表示，该场景表示是1）在几何形状，外观和运动方面的视觉现实的高保真详细模型，2）在部分受控的实际环境中通过捕获可机器学习，3）语义参数化以允许简单直观的编辑，4）快速可视化以进行实时探索。基于这种表示，我们将开发场景处理技术，允许个人创建，更改，探索和共享高保真虚拟对象和场景，解锁增强/虚拟现实，游戏，产品设计，制造，教育，机器人和医疗领域的一系列全新应用。

Digital representations of visual reality and imagination is an integral part of almost all scientific disciplines, industry, and society. Imaging techniques and computer graphics have successfully solved the problems of creating accurate projections, e.g. image and video, of the real world over the last decades. However, such projections are fundamentally limited representations of the underlying scenes and only allow for passive consumption such as viewing on a screen. Many applications from image editing to augmented/virtual reality instead require active 3D exploration, creation, and editing of scenes, for which we need full virtual scene models. Creating virtual scenes that are high fidelity models of the real world and beyond via digital modelling or capture has traditionally been a privilege only available to corporations with educated artists and engineers, working with complex software and hardware tools for countless hours. These professionals produce and work with carefully designed parametrizations of geometry, appearance, and motion, which allow them to author and edit virtual scenes with all the intricate details of reality and their imagination. On the other end, many computer vision techniques try to have a shortcut by capturing scenes from the real world via simpler sensors in uncontrolled environments, or altering images of scenes with algorithms that only implicitly encode scene semantics, e.g. in latent spaces of artificial neural networks. These lead to scene representations of lower quality and in a form that is not easily editable for semantically meaningful modelling. The objective of this research project is to tackle these shortcomings and develop a scene representation that is 1) a high fidelity detailed model of visual reality in terms of geometry, appearance, and motion, 2) machine learnable via capture in partially controlled practical environments, 3) semantically parametrized to allow for easy and intuitive edits, 4) fast to visualize for real-time exploration. Based on this representation, we will develop scene processing techniques that will allow individuals to create, alter, explore, and share high fidelity virtual objects and scenes, unlocking a completely new set of applications in augmented/virtual reality, gaming, product design, manufacturing, education, robotics, and medical domains.