CAREER: Modeling Situated Intention during Nondeterministic Pedestrian-Vehicle Interactions through Explainable Compositional Learning of Naturalistic Driving Data

随着人工智能的快速发展，自动化水平更高的车辆正在进入日常生活。自动驾驶技术有望改善交通安全，提高出行效率，保护环境，减少老一辈或残疾人的行动障碍，从而带来整体的社会效益。然而，嵌入自动驾驶汽车的现有算法在识别道路和人行道上快速变化的行人意图方面仍然面临着根本性的挑战，这使得预测他们的行为和规划车辆运动变得困难。这些限制阻碍了在城市环境中实施完全自动和安全的汽车，并给行人和其他道路使用者带来了额外的风险。这个项目的重点是开发新的技术来模拟和预测行人的复杂和不断变化的意图。通过学习司机的思维过程和他们的驾驶反应，将创建一种算法，为自动汽车配备类似的能力，以顺利和安全地与行人和其他道路使用者互动。研究过程将包括自然驾驶数据收集、主题实验、知识建模和学习算法开发。开发的算法将在身临其境的虚拟环境中进行评估。该项目还包括在工程教育中促进以用户为中心的设计的活动，培养与人工智能技术相关的偏见和伦理问题的意识，并增加科学、技术、工程和数学领域代表性不足的社区的参与。该项目克服了目前行人行为预测的局限性，实现了自动驾驶车辆与行人之间的相互可理解。主要不同于传统的静态的关键时刻行人意图的观点，本研究考察了动态(变化)和交互情境下行人非言语行为与意图变化之间的关系。该项目通过基于事件分割的视频实验，同时收集时间视频片段和人类推理描述。然后，提出了一种构件式学习方法，将语言特征和视觉特征结合起来进行学习。该方法通过从普通驾驶员生成集合特征来避免专家选择特征空间的刚性结构，并且学习模型的三级可解释性可以从输入特征中调整模型输出。最后，通过在虚拟交互式行人模拟器上的主题实验对所开发的意图预测模型进行了评估。研究成果将通过行业合作者和会议分享，行人行为基准数据集将向公众传播。研究成果将被纳入工程教育，以促进考虑到用户的感受、价值观和整体心理状态的设计方法(移情设计)。这些教育活动将包括在调查员所在的大学、自动驾驶研究社区和行业的课程。他们将提高人们对人工智能设计中以人为中心的关键问题的认识，如偏见、信任和社交智能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

With the fast progress of artificial intelligence, vehicles with higher levels of automation are entering daily life. Automated driving technologies are expected to improve traffic safety, promote travel efficiency, protect the environment, reduce mobility barriers for older generations or people with disabilities, and thus deliver overall societal benefits. However, existing algorithms embedded in autonomous vehicles still face fundamental challenges in recognizing the quickly changing intentions of pedestrians moving on the road and sidewalks, making it hard to predict their behavior and plan vehicle motions. Such limitations impede the implementation of fully autonomous and safe cars in city environments and create additional risks for pedestrians and other road users. This project focuses on developing novel techniques to model and predict the complex and changing intentions of pedestrians. By learning the thinking process of drivers and their driving responses, an algorithm will be created to equip the automated cars with similar capabilities to interact with pedestrians and other road users smoothly and safely. The research process will include naturalistic driving data collection, subject experiments, knowledge modeling, and learning algorithm development. The developed algorithm will be evaluated in an immersive virtual environment. The project also includes activities to promote user-centered design in engineering education, foster the awareness of biases and ethical issues related to artificial intelligence technologies, and increase the participation of underrepresented communities in Science, Technology, Engineering, and Mathematics. This project surmounts limitations of current pedestrian behavior prediction to achieving mutual intelligibility between autonomous vehicles and pedestrians. Principally, unlike the traditional static view of pedestrian intention at a critical moment, this research investigates the relationship between non-verbal actions and intention changes of pedestrians moment-to-moment in dynamic (changing) and interactive situations. The project collects temporal video segments and human reasoning descriptions simultaneously through event-segmentation-based video experiments. Then, it develops a compositional learning method to learn and combine language features with visual features. This method can avoid the rigid structure of expert-selected feature space by creating collective features from ordinary drivers, and the three-level explainability of the learning model can justify model outputs from input features. Finally, the developed intention prediction model will be evaluated through subject experiments in a virtual interactive pedestrian simulator. The research findings will be shared through industrial collaborators and conferences, and a pedestrian behavior benchmark dataset will be disseminated to the public. The research results will be included in engineering education to promote design approaches that take into account the users’ feelings, values, and overall mental state (empathic design). These educational activities will include courses at the investigator’s university, the autonomous driving research community, and industry. They will increase the awareness of critical human-centered AI issues like biases, trust, and social intelligence in design of AI.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.