场景理解是一个在数字城市构建、机器人导航、无人驾驶和虚拟现实等诸多领域有着广泛应用的重要研究方向。虽然有着多年的研究，但由于三维重建技术的不成熟，解析复杂的场景仍是一个难点问题。为此，本项目提出了一种基于深度和图像信息融合的三维场景理解的新方法。利用深度图像与可见光图像的互补性，定义了判别性高且对光照变化、遮挡等干扰信号鲁棒的三维低层基元特征；在匹配过程中，加入了中层语义信息和可变先验模型约束，并通过求解一个有约束的二次规划问题实现三维场景匹配和姿态估计，最终实现完整的场景重建。在此基础上，利用分类学习的方法识别地形、根据几何信息的推理检测障碍物和匹配三维场景实现场景识别，有效地解决了四足机器人平稳落足点的选择、群机器人的自主避障和移动机器人的自主导航和定位等难点问题。

Scene understanding has wide applications in many fields, ranging from digital city construction, robot navigation, autonomous driving to virtual reality. In spite of numerous efforts in the past decade, understanding complex scenes remains an unsolved challenge due to lacking efficient 3D reconstruction method. This project proposes a new method towards understanding full 3D natural scenes in real time. It relies essentially on fusing depth and visual image cues. We first extract basic features that are discriminative and highly robust to lighting and occlusions to match with each other. During the course of matching, we integrate middle-level contexts and adaptive priors to improve the matching rate. By viewing feature matching as a constrained quadratic programming problem, natural scenes are efficiently registered and 3D pose is correctly estimated. Following the results, we utilize machine learning tools to recognize terrain, rely on 3D geometry inference to detect obstacles and recognize scenes via 3D scene registration, which can effectively solve the challenging problems of foothold selection in a walking robot, obstacle avoiding in swarm robotics and SLAM in a mobile robot.