Collaborative Research: Enabling Machine Learning based Cooperative Perception with mmWave Communication for Autonomous Vehicle Safety

协作研究：通过毫米波通信实现基于机器学习的协作感知，以实现自动驾驶汽车安全

• 批准号: 2010332
• 负责人: Song Fu
• 金额: $ 24.65万
• 依托单位: University of North Texas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-15 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2010332&HistoricalAwards=false
• 关键词: Collaborative; Research; Enabling; Machine; Learning

By understanding what and how data are exchanged among autonomous vehicles, from a machine learning perspective, it is possible to realize precise cooperative perception on autonomous vehicles, enabling massive amounts of sensor information to be shared amongst vehicles. Such an advance can be extremely useful to extend the line of sight and field of view of autonomous vehicles, which otherwise suffers from blind spots and occlusions. The extended field of view on autonomous vehicles will be beneficial at times when there are occlusions preventing a complete perception of the environment. This increase in situational awareness promotes safe driving over a narrow scope and improves traffic flow efficiency over an extended scope. The proposed research work will not only change the way people think about the perception system on autonomous vehicles but could also open up opportunities to design novel systems that were previously inconceivable. This project offers a wide variety of research activities from data collection, algorithm design, system development, and in-the-field evaluation, which will be attractive to students with various backgrounds and goals. Undergraduate and graduate students will be involved directly in the research activities as assistants at different levels. The expected research outcomes from this project will also enhance the current curricula related to machine learning, Internet of things, and wireless communications.The main research objective of this project is to understand the sensing and communication challenges to achieving cooperative perception among autonomous vehicles, and to use the insights thus gained to guide the design of suitable data exchange format, data fusion algorithms, and efficient millimeter wave vehicular communications. Results from this project will include a machine learning based cooperative perception framework, which will shed light on effectively combining feature maps, derived from machine learning models on autonomous vehicles, in a distributed manner. The resulted feature map compression and feature map selection approaches will significantly reduce the amount of data exchanged among vehicles, enabling agile and precise cooperative perception on connected and autonomous vehicles. The proposed scalable feature map transmission mechanism jointly considers the application requirements, link and physical layer characteristics of millimeter wave links, enabling sensor data sharing on a massive scale among autonomous vehicles. The implemented system and evaluation platform will serve as a convincing proof-of-concept for the proposed solution, thus opening the door to widespread adoption of cooperative perception applications via millimeter wave communications in future vehicle networks. The collected dataset from this project will be made publicly available, serving as a catalyst for enabling innovative research on cooperative object detection, vehicular edge computing, and machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

从机器学习的角度来看，通过了解自动驾驶汽车之间交换数据的内容和方式，可以实现自动驾驶汽车的精确协同感知，从而使大量传感器信息能够在车辆之间共享。这一进步对于延长自动驾驶汽车的视线和视野非常有用，否则会受到盲点和闭塞的困扰。自动驾驶汽车的扩展视野在有遮挡的情况下是有益的，无法完全感知环境。这种态势感知的增加促进了窄范围内的安全驾驶，并提高了大范围内的交通流量效率。拟议中的研究工作不仅会改变人们对自动驾驶汽车感知系统的看法，还可能为设计以前无法想象的新系统提供机会。这个项目提供了广泛的研究活动，从数据收集，算法设计，系统开发和实地评估，这将吸引不同背景和目标的学生。本科生和研究生将作为不同层次的助理直接参与研究活动。该项目的预期研究成果也将增强当前与机器学习、物联网和无线通信相关的课程。该项目的主要研究目标是了解实现自动驾驶车辆之间协同感知的传感和通信挑战，并利用由此获得的见解指导设计合适的数据交换格式、数据融合算法和高效的毫米波车辆通信。该项目的成果将包括一个基于机器学习的合作感知框架，该框架将阐明如何以分布式方式有效地结合自动驾驶汽车上的机器学习模型生成的特征图。由此产生的特征图压缩和特征图选择方法将大大减少车辆之间交换的数据量，从而实现对联网和自动驾驶车辆的敏捷和精确的协同感知。所提出的可扩展特征图传输机制综合考虑了毫米波链路的应用需求、链路和物理层特性，实现了自动驾驶汽车间传感器数据的大规模共享。实施的系统和评估平台将作为拟议解决方案的令人信服的概念验证，从而为通过毫米波通信在未来车辆网络中广泛采用协作感知应用打开大门。从该项目收集的数据集将公开提供，作为实现协作目标检测、车辆边缘计算和机器学习方面的创新研究的催化剂。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Adaptive Vehicle Platooning with Joint Network-Traffic Approach

• DOI: 10.1109/globecom46510.2021.9685116
• 发表时间: 2021-12
• 期刊: 2021 IEEE Global Communications Conference (GLOBECOM)
• 作者: Chinmay Mahabal;Hua Fang;Honggang Wang;Qing Yang

CoConv: Learning Dynamic Cooperative Convolution for Image Recognition

• DOI: 10.1109/icme51207.2021.9428105
• 发表时间: 2021-07
• 期刊: 2021 IEEE International Conference on Multimedia and Expo (ICME)
• 作者: Kien X. Nguyen;Tiffany Ryu;Jocelyn Zhang;Xu Ma;Qing Yang;Song Fu;P. Palacharla;N. Wang;Xi Wang

Vehicular Edge Computing for Multi-Vehicle Perception

用于多车辆感知的车辆边缘计算

• DOI: 10.1109/metrocad51599.2021.00011
• 发表时间: 2021
• 期刊: 2021 IEEE International Conference on Connected and Autonomous Driving
• 作者: Tang, Sihai;Gu, Zhaochen;Fu, Song;Yang, Qing
• 通讯作者: Yang, Qing

VECFrame: A Vehicular Edge Computing Framework for Connected Autonomous Vehicles

VECFrame：用于联网自动驾驶车辆的车辆边缘计算框架

• DOI: 10.1109/edge53862.2021.00019
• 发表时间: 2021
• 期刊: 2021 IEEE International Conference on Edge Computing (EDGE
• 作者: Tang, Sihai;Chen, Bruce;Iwen, Harold;Hirsch, Jason;Fu, Song;Yang, Qing;Palacharla, Paparao;Wang, Nannan;Wang, Xi;Shi, Weisong
• 通讯作者: Shi, Weisong

Spatial Pyramid Attention for Deep Convolutional Neural Networks

深度卷积神经网络的空间金字塔注意力

• DOI: 10.1109/tmm.2021.3068576
• 发表时间: 2021
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Ma, Xu;Guo, Jingda;Sansom, Andrew;Mcguire, Mara;Kalaani, Andrew;Chen, Qi;Tang, Sihai;Yang, Qing;Fu, Song
• 通讯作者: Fu, Song