Understanding the world behind the image

长期以来，实际的计算机视觉应用一直局限于“机器视觉”领域：用于装配线上的自动检测和分析的图像处理技术。在过去的十年里，随着超大规模图像数据集、巨大的计算能力和强大的深度学习算法的出现，我们见证了计算机视觉在大规模消费应用中的出现。例如，计算机已经学会了将虚拟对象组合成用于增强现实的真实视频馈送，逼真地再现视频中的人的外表，通过检测障碍物的位置和方向来安全地驾驶自动驾驶汽车，以及从一张图像中稳健地估计场景的深度-有效地理解3D世界。 尽管取得了这些进步，但计算机视觉系统仍面临着一个主要限制：它们难以理解真实世界。大多数技术估计世界的单个组成部分(照明条件、场景深度、对象的几何图形、曲面的方向等)。同时，不考虑当图像形成时所有这些组件都在发挥作用的事实。事实上，图像是通过光与场景中表面和对象的几何体和反射率之间的一系列复杂相互作用来创建的。虽然人眼可以通过数百万年的适应很容易地理解这些效果的组合(例如，我们很容易解释阴影是由被遮挡的明亮光源产生的)，但对于数码相机和对其图像进行操作的算法则不能说同样的话(例如，阴影可能被误解为对象)。 这项研究计划将引入新的方法，以全面的方式自动理解场景的3D、照明和反射属性。为此，我们将实现以下四个目标。我们将：1)引入新的算法来估计场景中空间变化的照度；2)开发一种新的方法来使用便携式多光捕获设备来有效地估计表面和场景的空间变化的反射特性；3)结合成像的物理模型，以从一幅图像中联合估计光照、反射比和几何形状，以推动整体场景理解的最新进展；以及4)捕获新的真实世界照明和表面反射比的数据库，以忠实地数字化和重建尺度上的世界。 除了上述应用，我们提议的活动还将影响其他领域，如计算机图形、视频游戏、电影、虚拟和增强现实以及人工智能。拉瓦尔大学的这一研究项目将通过其关键的技术贡献，并通过培训高素质的开发人员，为保持甚至扩大加拿大在这些领域的相关性做出贡献。

Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.