Engineering Robust 3D Representations from Robotic Visual Sensors for Navigation & Scene Analysis

无论我们是走路还是开车，我们都在不断地制作心理地图，以确定路径/道路，地标，物体和其他人类代理的位置以及它们之间的关系。 这些心理地图和模型使我们能够导航到一个特定的位置，开车去上班，修剪草坪或清洁我们的家。 当我们推理或计划某些活动时，比如修理篱笆或粉刷房子，我们也会构建心理地图。 将构建这种思维地图的能力转移给机器人是我们研究的动力。 人类主要是用眼睛来做这件事的。 如果我们能为一个带摄像头的机器人做同样的事情呢？ 然后，机器人将能够理解和推理世界，以便执行任务。 心理地图只是时间的快照。 该地图还必须包括区分移动实体与静态地标和占位符。 根据手头的任务，详细程度会有所不同。 静态组件的完整3D重建与分析、重新设计或可能的3D打印对象和环境相关。 用于构建3D地图的两种视觉技术包括SFM（运动结构）和SLAM（同时定位和映射）。 这两种方法非常相似，区别在于SFM通常是离线的，而SLAM是在线的。 这两种方法都包括一个前端，它检测感兴趣的特征，并使用摄影测量来关联视图之间的这些数据点。 这两种方法的后端都是最小化重投影误差的优化方法。这两个过程可能容易或困难，具体取决于所使用的传感器、环境的复杂程度和所需的性能。 自动驾驶汽车受益于使用提供精确环境测量的LIDAR传感器和提供位置信息的GPS。 然而，对于许多真实的世界应用，仅仅依靠视觉SLAM/SFM是不健壮的。 SLAM/SFM解决方案在摄像机出现锐角时可能非常脆弱，例如在办公楼中，姿势跟踪失败，视图无法注册。 这里严重依赖于特征点数据关联&缺少特征或高度重复的特征可能会产生问题。 SLAM/SFM本质上是几何问题，而深度学习神经网络最近在视觉感知方面取得了巨大的成功，特别是在对场景标签进行分类方面。 深度学习的场景标签可以指导特征点关联以及几何、光度和其他一致性检查，以帮助实现视觉SLAM/SFM鲁棒性。 感兴趣的问题包括（1）使廉价的导航机器（诸如仅具有相机系统的工业清洁器）自动化;（2）构建用于市政基础设施监测（诸如用于道路、桥梁）的历史地图;以及其他。 一个强大的SLAM/SFM解决方案可以帮助解决这些应用和其他应用。

Whether we are walking or driving, we are constantly making mental maps to determine where the pathway/road, landmarks, objects and other human agents are situated and their relationships to each other. These mental maps and models are what allow us to navigate to a particular location, drive to work, mow our lawns or clean our homes. We also build mental maps when we reason or plan certain activities like repairing a fence or painting a home. Transferring the ability to build such a mental map for a robot is what drives our research. Humans are able to chiefly do this with our eyes. What if we could do the same for a robot with a camera? The robot would then be able to understand and reason in the world so as to perform a task. A mental map is just a snapshot in time. This map has to also include differentiating moving entities from static landmarks and placeholders. Depending on the task at hand, the level of detail will vary. A complete 3D reconstruction of the static components is relevant for analysis, re-engineering or possibly 3D printing the objects and environments. Two visual techniques that are used to build 3D maps include SFM (Structure from Motion) and SLAM (Simultaneous Localization And Mapping). The two methods are very similar, the differentiating factor being that SFM is typically off-line while SLAM is online. Both methods include a front end which detects features of interest and uses photogrammetry to associate these data points between views. The back-end for both methods is an optimization method that minimizes re-projection errors. Both processes may be easy or difficult depending on the sensors being used, the level of complexity in the environment and the required performance. Autonomous automobiles benefit from using a LIDAR sensor which provides precise environmental measurements and a GPS which provides location information. However, for many real world applications, just relying on visual SLAM/SFM is not robust. SLAM/SFM solutions can be very brittle when the camera makes sharp corners such as in an office building where the tracking of pose fails and views cannot be registered. There is a heavy reliance on the feature point data association & a lack of features or highly repetitive features can be problematic. SLAM/SFM are essentially problems of geometry whereas deep learning neural networks have had recent immense success in visual perception, especially in categorizing scene labels. Deep Learning's scene labels can guide feature point associations as well as geometrical, photometric & other consistency checks to help make visual SLAM/SFM robust. Problems of interest include (1) automating inexpensive navigating machines such as industrial cleaners with only a camera system; (2) building historical maps for municipal infrastructure monitoring such as for roads, bridges; and others. A robust SLAM/SFM solution can help solve these applications and others.