Promoting 3D Environment Perception Ability of Autonomous Systems

从图像序列进行环境感知和建模是动态环境下智能车辆的一项具有挑战性的任务。作为视觉感知的重要组成部分，如何有效地对世界进行建模和表示是近几十年来的一个活跃研究领域。尽管在理论和实践上都取得了长足的进展，但环境感知问题仍然远远没有得到解决，主要是因为三个关键挑战：(1)如何恢复包含刚性、非刚性、铰接和运动物体的动态自然场景的三维结构；(2)如何在实际导航场景中高效而稳健地表示工作环境；(3)如何提高机器学习模型的泛化能力。这些问题非常复杂，因此很少在文献中得到很好的解决。该项目的主要目标是通过开发一种高效和健壮的动态自然环境表示来解决这些挑战，并促进智能系统的3D感知能力。具体地说，我们将在这个项目中调查三个关键的研究问题。第一个问题是场景深度估计和语义分割。提出了一种基于多任务学习的深度估计和语义分割的统一学习模型。该模型综合了序列的几何约束和时间信息，能够同时恢复深度信息和语义信息，具有较高的精度。第二个问题是语义对象检测。我们提出了一种新的目标检测学习模型，该模型能够模仿人脑从训练样本中进行推理和推理。开发了一种新的学习范式，通过主动学习和强化学习来提高模型的泛化能力。对于第三个问题，我们提出了一种新的统一框架来表示动态环境，通过利用场景中的所有信息来形成混合表示。这项研究通过整合来自现场的所有信息，为环境感知提供了一个新的视角。特别是，该项目强调了深度估计、语义分割、对象检测和3D映射等问题的新算法方面。该方法使得自主系统在未知动态环境中导航的环境映射更加高效和有效。这项研究不仅具有重要的学术意义，而且也是汽车和机器人行业迫切需要的。这项研究是朝着调查者的长期研究努力迈出的基本一步：通过视觉线索建立高度智能的系统。我们还将把拟议的框架和研究成果纳入我们的教育和外展活动，以改进现有的课程课程，并吸引更多的学生，特别是那些来自代表人数较少的少数族裔的学生，在计算机科学和工程领域工作。

Environment perception and modeling from image sequences is a challenging task of intelligent vehicles in dynamic environments. As a critical component of visual perception, how to efficiently model and represent the world has been an active research area during the past decades. Although considerable progress has been made both in theory and in practice, the problem of environment perception remains far from being solved, owing primarily to three critical challenges: (i) how to recover the 3D structure of dynamic natural scenes coupled with rigid, nonrigid, articulated, and moving objects; (ii) how to efficiently and robustly represent the working environment in practical navigation scenarios; and (iii) how to increase the generalization ability of a machine learning model. These issues are very complex and have thus seldom been well-addressed in the literature. The main objective of this project is to solve these challenges and promote the 3D perception ability of intelligent systems by developing an efficient and robust representation of dynamic natural environments. Specifically, we will investigate three critical research problems in this project. The first one is on scene depth estimation and semantic segmentation. We propose a unified learning model for depth estimation and semantic segmentation using multi-task learning. By integrating the geometric constraint and the temporal information of the sequence, the proposed model can recover the depth and semantic information simultaneously with high accuracy. The second problem is semantic object detection. We propose a new learning model for object detection with the ability to make reasoning and deduction from training examples by mimicking human brains. A new learning paradigm is developed to increase the model generalizability via active and reinforcement learning. For the third problem, we propose a novel unified framework to represent dynamic environments by exploring all information from the scene to form a hybrid representation. The study envisions a novel perspective for environment perception by integrating all information from the scene. In particular, the project emphasizes new algorithmic aspects for problems of depth estimation, semantic segmentation, object detection, and 3D mapping. The proposed technique makes the environment mapping more efficient and effective for autonomous systems to navigate in unknown dynamic environments. This research is not only academically significant but also urgently needed by automotive and robotic industries. This research is a fundamental step towards the investigator's long-term research endeavor: building a highly intelligent system through visual cues. We will also integrate the proposed framework and research results into our education and outreach activities to enhance existing course curriculums and attract more students, especially those from underrepresented minorities, to work in computer science and engineering.