Promoting 3D Environment Perception Ability of Autonomous Systems

基于图像序列的环境感知和建模是智能车辆在动态环境中的一项具有挑战性的任务。作为视觉感知的重要组成部分，如何有效地对世界进行建模和表征一直是近几十年来研究的热点。尽管在理论和实践方面都取得了相当大的进展，但环境感知问题仍远未解决，主要原因是三个关键挑战：(i)如何恢复动态自然场景的3D结构，这些场景与刚性、非刚性、铰接和移动物体相结合；（ii）如何在实际航行场景中有效和稳健地表示工作环境；（三）如何提高机器学习模型的泛化能力。这些问题非常复杂，因此很少在文献中得到很好的解决。该项目的主要目标是解决这些挑战，并通过开发动态自然环境的高效和鲁棒表示来提高智能系统的3D感知能力。具体来说，我们将在这个项目中调查三个关键的研究问题。第一部分是场景深度估计和语义分割。我们提出了一种基于多任务学习的深度估计和语义分割的统一学习模型。该模型通过整合序列的几何约束和时间信息，可以同时恢复深度信息和语义信息，具有较高的精度。第二个问题是语义对象检测。我们提出了一种新的目标检测学习模型，该模型具有模仿人脑从训练样本中进行推理和演绎的能力。提出了一种新的学习范式，通过主动学习和强化学习来提高模型的可泛化性。对于第三个问题，我们提出了一个新的统一框架来表示动态环境，通过探索来自场景的所有信息来形成混合表示。该研究通过整合来自场景的所有信息，设想了一种新的环境感知视角。该项目特别强调了深度估计、语义分割、目标检测和3D映射等问题的新算法方面。该技术使自主系统在未知动态环境中导航的环境映射更加高效。这一研究不仅具有重要的学术意义，而且也是汽车和机器人行业的迫切需要。这项研究是研究者长期研究努力的基础一步：通过视觉线索建立一个高度智能的系统。我们亦会把建议的架构和研究成果纳入我们的教育和外展活动，以改善现有的课程，吸引更多学生，特别是少数族裔学生，从事计算机科学和工程方面的工作。

Environment perception and modeling from image sequences is a challenging task of intelligent vehicles in dynamic environments. As a critical component of visual perception, how to efficiently model and represent the world has been an active research area during the past decades. Although considerable progress has been made both in theory and in practice, the problem of environment perception remains far from being solved, owing primarily to three critical challenges: (i) how to recover the 3D structure of dynamic natural scenes coupled with rigid, nonrigid, articulated, and moving objects; (ii) how to efficiently and robustly represent the working environment in practical navigation scenarios; and (iii) how to increase the generalization ability of a machine learning model. These issues are very complex and have thus seldom been well-addressed in the literature. The main objective of this project is to solve these challenges and promote the 3D perception ability of intelligent systems by developing an efficient and robust representation of dynamic natural environments. Specifically, we will investigate three critical research problems in this project. The first one is on scene depth estimation and semantic segmentation. We propose a unified learning model for depth estimation and semantic segmentation using multi-task learning. By integrating the geometric constraint and the temporal information of the sequence, the proposed model can recover the depth and semantic information simultaneously with high accuracy. The second problem is semantic object detection. We propose a new learning model for object detection with the ability to make reasoning and deduction from training examples by mimicking human brains. A new learning paradigm is developed to increase the model generalizability via active and reinforcement learning. For the third problem, we propose a novel unified framework to represent dynamic environments by exploring all information from the scene to form a hybrid representation. The study envisions a novel perspective for environment perception by integrating all information from the scene. In particular, the project emphasizes new algorithmic aspects for problems of depth estimation, semantic segmentation, object detection, and 3D mapping. The proposed technique makes the environment mapping more efficient and effective for autonomous systems to navigate in unknown dynamic environments. This research is not only academically significant but also urgently needed by automotive and robotic industries. This research is a fundamental step towards the investigator's long-term research endeavor: building a highly intelligent system through visual cues. We will also integrate the proposed framework and research results into our education and outreach activities to enhance existing course curriculums and attract more students, especially those from underrepresented minorities, to work in computer science and engineering.