Understanding the world behind the image

长期以来，实际的计算机视觉应用一直局限于“机器视觉”领域：用于自动检测和分析的图像处理技术。在过去的十年中，随着超大型图像数据集、强大的计算能力和强大的深度学习算法的出现，我们见证了计算机视觉在大规模消费者应用中的出现。例如，计算机已经学会了将联合收割机的虚拟物体组合成增强现实的真实的视频，逼真地再现视频中人的外观，通过检测障碍物的位置和方向来安全地驾驶自动汽车，以及从单个图像中稳健地估计场景的深度--有效地理解3D世界。 尽管取得了这些进展，但计算机视觉系统仍受到一个主要限制：它们很难理解整个真实的世界。大多数技术估计世界的单个组件（照明条件，场景深度，对象的几何形状，表面的方向等）。而不考虑当图像形成时所有这些成分都在起作用的事实。事实上，图像是通过光与场景中表面和物体的几何形状和反射率之间的一系列复杂相互作用创建的。虽然人眼可以很容易地理解这些影响的组合，通过数百万年的适应（例如，我们很容易理解阴影是由被遮挡的亮光源产生的），对于数字照相机和对它们的图像进行操作的算法则不能这样说（例如，阴影可能被误解为对象）。 该研究计划将介绍以整体方式自动理解场景的3D，照明和反射特性的新方法。为此，我们将努力实现以下四个目标。我们将：1）引入用于估计场景中的空间变化的照明的新算法; 2）开发使用便携式多光捕获设备来有效地估计表面和场景的空间变化的反射率属性的新颖方法; 3）结合图像形成的物理模型以用于从单个图像联合估计照明、反射率和几何形状，以推动整体场景理解的现有技术;以及4）捕获真实世界照明和表面反射率的新数据库，以忠实地再现和按比例重建世界。 除了上述应用外，我们的活动还将影响其他领域，如计算机图形，视频游戏，电影，虚拟和增强现实以及人工智能。拉瓦尔大学的这项研究计划将有助于通过其关键技术贡献，并通过培训高素质的人才来维持甚至扩大加拿大在这些领域的相关性。

Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.