Understanding the world behind the image

长期以来，实际的计算机视觉应用发现自己局限于“机器视觉”领域：用于装配线自动检查和分析的图像处理技术。在过去的十年中，随着大型图像数据集、巨大的计算能力和强大的深度学习算法的出现，我们见证了计算机视觉在大规模消费者应用中的出现。例如，计算机已经学会了将虚拟物体结合到增强现实的真实视频中，在视频中逼真地重现人的外观，通过检测障碍物的位置和方向来安全驾驶自动驾驶汽车，以及从单个图像中稳健地估计场景的深度——有效地理解3D世界。尽管取得了这些进步，但计算机视觉系统仍有一个主要的局限性：它们难以从整体上理解现实世界。大多数技术一次估计世界的单一组成部分（照明条件，场景深度，物体的几何形状，表面的方向等），而没有考虑到当图像形成时所有这些组成部分都在发挥作用。事实上，图像是通过光与场景中物体表面的几何和反射率之间的一系列复杂相互作用而产生的。通过数百万年的适应，人眼可以很容易地理解这些效果的组合（例如，我们可以很容易地解释阴影是由明亮的光源被遮挡造成的），但数码相机和对其图像进行操作的算法却不能这样说（例如，阴影可能被误解为物体）。本研究计划将引入新的方法，以整体的方式自动理解场景的3D，照明和反射特性。为此，我们将努力实现以下四个目标：我们将：1)介绍用于估计场景中空间变化照明的新算法；2)开发一种新的方法，利用便携式多光捕获装置有效地估计表面和场景的空间变化反射率特性；3)结合图像形成的物理模型，从单个图像中共同估计照明，反射率和几何形状，以推动整体场景理解的最新技术；4)获取真实世界照明和表面反射率的新数据库，以忠实地数字化和大规模重建世界。除了上述应用之外，我们提出的活动还将影响其他领域，如计算机图形学、视频游戏、电影、虚拟和增强现实以及人工智能。拉瓦尔大学的这个研究项目将通过其关键的技术贡献，并通过培养高素质的人才，有助于保持甚至扩大加拿大在这些领域的相关性。

Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.