Deep Visual Geometry Machines

视觉几何是从图像中理解环境并估计观察者位置的过程。它是自动驾驶汽车和送货无人机等自动驾驶车辆(AV)的核心，并决定着机器自动导航和与世界互动的能力。对于增强现实(AR)和混合现实(MR)来说，视觉几何是决定一个人如何将虚拟世界与现实世界混合在一起的真实感和稳定性的因素之一。因此，发展视觉几何对于将这些新技术带入我们的日常生活是必不可少的。*然而，尽管它很重要，但现有的视觉几何方法还不够准确，不足以在野外部署。性能有限的主要原因是，在视觉几何方面，我们仍然依赖传统方法，而不是计算机视觉的其他领域，在深度学习的帮助下，机器在许多任务上都超过了人类。此外，当前最先进的基于深度学习的视觉几何方法在其他应用程序中的表现也不如其他应用程序，例如识别图像。我认为造成这种对比的主要原因是，通常情况下，对于深度学习带来令人难以置信的成功的应用程序，问题在数学上被过度确定，而视觉几何由于在将3D现实世界的数据投影到2D图像时丢失深度信息而被低估。*为了克服这个问题，我的目标是建立深度视觉几何机器，这是深度网络的集合，专门用于视觉几何管道的每一项任务：从图像中提取关键信息，找到图像之间的对应关系，并通过对应关系恢复观察者的状态。通过将每个网络限制为特定的子任务，我们可以将视觉几何问题描述为一组超定的子问题，深度学习已经证明擅长解决这一问题。通过这样做，我预计观察者估计位置的准确性以及环境地图的准确性将显著提高，可能超过人类。*拟议的研究将立即对许多应用产生广泛影响，包括但不限于自动驾驶、交付无人机、家用机器人、增强现实和混合现实。此外，拟议中的技术将使机器能够学习规划和行动，从而允许更复杂的应用。例如，这可能包括允许自动驾驶车辆和无人机在加拿大北部的偏远地区运行。*目前的提议还有望有助于在机器学习和计算机视觉领域对HQP进行培训。目前，这两个地区在加拿大和世界范围内对HQP的需求都很高。

Visual Geometry is the process of understanding the environment and estimating the position of the observer from images. It is at the core of Autonomous Vehicles (AV) such as self-driving cars and delivery drones, and determines how well a machine can automatically navigate and interact with the world. For Augmented Reality (AR) and Mixed Reality (MR), Visual Geometry is one of the deciding factors on how realistic and stable one can mix the virtual world with the real world. Thus, advancing Visual Geometry is essential in bringing these new technologies into our daily lives.******However, despite its importance, existing methods for Visual Geometry is not accurate enough to be deployed in the wild. The main reason for the limited performance is that we still rely on traditional methods when it comes to Visual Geometry, in contrast to other areas in Computer Vision, where machines now outperform humans in many tasks through the help of Deep Learning. Moreover, the current state- of-the-art Deep Learning-based methods for Visual Geometry are not performing as well as in other applications, for example recognizing images. I argue that the main reason for this contrast is that typically, for the applications where Deep Learning brought incredible success, the problem is mathematically over-determined, whereas Visual Geometry is under-determined due to the loss of depth information when projecting 3D real-world data into 2D images.******To overcome this problem, I aim towards building the Deep Visual Geometry Machine, a collection of Deep Networks dedicated to each task of the Visual Geometry pipeline: extracting key information from images, finding correspondences between images, and recovering the state of the observer through correspondences. By limiting each network to a specific sub task, we can formulate the Visual Geometry problem as a collection of over-determined sub problems, which Deep Learning has shown to be good at solving. By doing so, I expect a significant improvement in the accuracy of the estimated position of the observer, as well as the mapping of the environment, possibly surpassing the humans.******The proposed research will have immediate broad impact on many applications including, but not limited to, autonomous driving, delivery drones, domestic robots, augmented reality, and mixed reality. Furthermore, the proposed technology would enable machines to learn to plan and act, thus allowing even more complicated applications. This could include, for example, allowing autonomous vehicles and drones to operate in remote regions in the north of Canada.******The current proposal also is expected to contribute to the training of HQP in the area of Machine Learning and Computer Vision. Currently, both areas are in high demand of HQP in Canada and world-wide.