CAREER: Modeling Situated Intention during Nondeterministic Pedestrian-Vehicle Interactions through Explainable Compositional Learning of Naturalistic Driving Data

随着人工智能的快速进步，自动化程度更高的车辆正在走进人们的日常生活。自动驾驶技术有望改善交通安全、提高出行效率、保护环境、减少老一辈或残疾人的出行障碍，从而带来整体社会效益。然而，嵌入自动驾驶汽车中的现有算法在识别道路和人行道上移动的行人快速变化的意图方面仍然面临着根本性挑战，这使得预测他们的行为和规划车辆运动变得困难。这些限制阻碍了全自动和安全汽车在城市环境中的实施，并给行人和其他道路使用者带来了额外的风险。该项目的重点是开发新技术来建模和预测行人复杂且不断变化的意图。通过学习驾驶员的思维过程及其驾驶反应，我们将创建一种算法，为自动驾驶汽车配备类似的功能，以便顺利、安全地与行人和其他道路使用者互动。研究过程将包括自然驾驶数据收集、主题实验、知识建模和学习算法开发。开发的算法将在沉浸式虚拟环境中进行评估。该项目还包括促进工程教育中以用户为中心的设计、培养对与人工智能技术相关的偏见和道德问题的认识以及增加代表性不足的社区对科学、技术、工程和数学的参与等活动。该项目克服了当前行人行为预测的局限性，实现自动驾驶车辆和行人之间的相互理解。原则上，与关键时刻行人意图的传统静态观点不同，本研究研究动态（变化）和交互情境中行人的非语言行为与时刻意图变化之间的关系。该项目通过基于事件分割的视频实验同时收集时间视频片段和人类推理描述。然后，它开发了一种组合学习方法来学习并将语言特征与视觉特征相结合。该方法可以通过创建普通驾驶员的集体特征来避免专家选择的特征空间的刚性结构，并且学习模型的三级可解释性可以证明输入特征的模型输出是合理的。最后，开发的意图预测模型将通过虚拟交互式行人模拟器中的主题实验进行评估。研究结果将通过行业合作者和会议分享，行人行为基准数据集将向公众传播。研究成果将纳入工程教育，以推广考虑用户感受、价值观和整体心理状态的设计方法（共情设计）。 这些教育活动将包括研究者所在大学、自动驾驶研究社区和行业的课程。 他们将提高人们对人工智能设计中的偏见、信任和社会智能等关键的以人为中心的人工智能问题的认识。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

With the fast progress of artificial intelligence, vehicles with higher levels of automation are entering daily life. Automated driving technologies are expected to improve traffic safety, promote travel efficiency, protect the environment, reduce mobility barriers for older generations or people with disabilities, and thus deliver overall societal benefits. However, existing algorithms embedded in autonomous vehicles still face fundamental challenges in recognizing the quickly changing intentions of pedestrians moving on the road and sidewalks, making it hard to predict their behavior and plan vehicle motions. Such limitations impede the implementation of fully autonomous and safe cars in city environments and create additional risks for pedestrians and other road users. This project focuses on developing novel techniques to model and predict the complex and changing intentions of pedestrians. By learning the thinking process of drivers and their driving responses, an algorithm will be created to equip the automated cars with similar capabilities to interact with pedestrians and other road users smoothly and safely. The research process will include naturalistic driving data collection, subject experiments, knowledge modeling, and learning algorithm development. The developed algorithm will be evaluated in an immersive virtual environment. The project also includes activities to promote user-centered design in engineering education, foster the awareness of biases and ethical issues related to artificial intelligence technologies, and increase the participation of underrepresented communities in Science, Technology, Engineering, and Mathematics. This project surmounts limitations of current pedestrian behavior prediction to achieving mutual intelligibility between autonomous vehicles and pedestrians. Principally, unlike the traditional static view of pedestrian intention at a critical moment, this research investigates the relationship between non-verbal actions and intention changes of pedestrians moment-to-moment in dynamic (changing) and interactive situations. The project collects temporal video segments and human reasoning descriptions simultaneously through event-segmentation-based video experiments. Then, it develops a compositional learning method to learn and combine language features with visual features. This method can avoid the rigid structure of expert-selected feature space by creating collective features from ordinary drivers, and the three-level explainability of the learning model can justify model outputs from input features. Finally, the developed intention prediction model will be evaluated through subject experiments in a virtual interactive pedestrian simulator. The research findings will be shared through industrial collaborators and conferences, and a pedestrian behavior benchmark dataset will be disseminated to the public. The research results will be included in engineering education to promote design approaches that take into account the users’ feelings, values, and overall mental state (empathic design). These educational activities will include courses at the investigator’s university, the autonomous driving research community, and industry. They will increase the awareness of critical human-centered AI issues like biases, trust, and social intelligence in design of AI.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.