Dense Monocular Reconstruction and Semantic Segmentation of 3D Environments

机器人学和增强现实(AR)的大多数应用都依赖于它们在其中运行的环境的某种形式的3D模型，以便与环境交互。在最基本的层面上，这些模型提供有关事物在环境中的位置的信息，使系统能够准确且安全地与环境交互。传统上，模型的质量和密度之间存在权衡(即，准确的模型通常由点云组成，没有关于点之间区域的信息)。这种稀疏性通常源于3D扫描仪的使用，例如激光。有一些扫描仪能够产生密集的信息，然而，这些扫描仪要么限于室内操作，要么具有极短的射程(通常为1-5米)。如果需要更有意义的交互，则需要描述环境内容的语义信息。快速生成准确的3D空间模型的能力具有各种影响，从增加自动化系统导航和探索世界的简易性，到改善AR生成的视觉与真实世界的交互。此外，还有潜在的残疾和无障碍应用，可以轻松快速地绘制建筑物或区域的地图，然后缩放打印的模型，以帮助视力受损的人导航。语义信息使系统能够理解世界并回答有关世界的问题。例如，有了语义信息，我们就可以问这样的问题：“这个房间里的椅子在哪里？”我们的目标是研究用于重建密集的环境3D模型以及生成这些模型的语义分割的系统。我们希望能够开发一种能够实时生成这些模型的系统，并最终在嵌入式和移动设备上生成这些模型，例如iPhone。此外，我们的目标是能够在当前基于传感器的系统无法生成这些模型的地方，即户外和超过5米的范围。为了改进现有的从单目图像重建3D模型的技术，我们计划利用卷积神经网络(CNN)，并结合现有的几何方法。然而，我们建议直接从网络计算完整的3D模型，而不是计算深度图像。虽然这一过程需要更多的内存，但我们相信这将允许更好地整合可用的信息。我们怀疑，直接使用3D信息将对语义分割特别重要，其中对象的某些视角可能会产生误导(例如，从上面看起来很像桌子的椅子)。我们还计划研究递归神经网络在一系列图像上提高重建质量的潜力，因为这条输入管道模仿了你在真实世界数据采集场景中最可能看到的那些。这个项目属于EPSRC信息和通信技术主题，特别是图像和视觉计算研究领域。

Most applications of robotics and augmented reality (AR) rely on some form of a 3D model of the environment in which they operate in order to interact with the environment. At the most basic level, these models provide information about where things are in the environment, allowing systems to accurately and safely interact with the environment. Conventionally there has been a trade-off between the quality and the density of the model (i.e. accurate models typically consist of a cloud of points with no information about the region between the points). This sparseness typically arises from the use of 3D scanners, such as lasers. There are scanners that are able to produce dense information, however, these scanners are either limited to indoors operation, or have an extremely short range (typically 1-5m). If more meaningful interaction is desired then semantic information, which describes the contents of the environment, is required.The ability to quickly generate accurate models of 3D spaces has a variety of impacts, from increasing the ease with which automated systems can navigate and explore the world, to improving the interaction of visuals generated by AR with the real world. Further, there are potential applications for disability and access, whereby buildings or areas can be easily and quickly mapped, and then scale models printed, to aid those with impaired vision to navigate ares. Semantic information allows systems to understand the world and answer questions about it. For example, with semantic information, we can ask questions like "Where are the chairs in this room?"We aim to investigate systems for both reconstructing dense 3D models of environments as well as generating semantic segmentations of those models. We hope to be able to develop a system that is capable of generating these models in real time, and ultimately on embedded and mobile devices, such as an iPhone. Further, we aim to be able to generate these models in places where current sensor based systems cannot, i.e. outdoors and over ranges greater than 5m. To improve on the existing techniques for reconstructing 3D models from monocular images, we plan to utilize convolutional neural networks (CNNs), alongside existing geometric methods. However, rather than computing a depth image, we propose to directly compute a full 3D model from the network. Although this process requires significantly more memory, we believe that this will allow for better integration of the available information. We suspect that the direct use of 3D information will be of particular importance for semantic segmentation, where certain viewing angles of objects can be misleading (e.g. a chair from above looks a lot like a table). We also plan to investigate the potential of recurrent neural networks to improve the quality of the reconstructions over a sequence of images, as this input pipeline mimics those that you would most likely see in real world data acquisition scenarios.This project falls within the EPSRC Information and Communication Technologies theme, specifically the Image and Vision Computing research area.