Understanding the world behind the image

点击翻译按钮获取中文摘要

Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.