Leveraging FPGAs for Machine Learning Implementation and Acceleration

这项研究计划提出开发新技术，利用机器学习来改进自动驾驶车辆的目标检测和计算机视觉。该技术将包括使用现场可编程门阵列(FGA)进行计算加速的软件和硬件。现场可编程门阵列是一种可编程的集成电路芯片，可在硬件中实现不同的应用。该计划包括三个相关的研究推动力：融合传感器融合技术的神经网络(NN)算法，用于现场可编程门阵列的NN电路架构，以及该技术的最终用户应用，例如在难以管理的条件下增强自动驾驶员的视力。 传统的自动车辆目标检测方法是一种改进的图像分类任务，它在定位帧中的目标(例如汽车、行人)的同时执行目标分类任务。然而，随着研究的进展，光探测和测距(LiDAR)数据的良好结果促进了对LiDAR和视觉(相机)传感器融合的研究。在这个研究项目中，我们建议探索和充分理解融合操作，如元素方式的平均，但以新的方式考虑单个传感器的预测性能。当一个传感器因天气、组件故障或入侵企图而受损时，传感器的性能评估就变得有用了。当传感器做出错误的预测时，融合预测网络必须能够动态补偿，以避免整个目标检测系统的退化。 用于自动驾驶车辆的最先进的计算机视觉解决方案越来越依赖于卷积神经网络(CNN)来获得高质量的结果。CNN通常有数百万个神经元和突触，这导致了很高的计算复杂性和存储要求。因此，在自动驾驶汽车和无人机上部署CNN作为对象探测器和传感器融合解决方案是困难的，因为这些平台的功率预算紧张，延迟要求低，计算资源稀缺。现场可编程门阵列已经成为为这些应用实现专用CNN加速器的流行衬底。 这一研究项目的成果将应用于通用汽车公司主办的多所大学举办的国际自动驾驶汽车大赛。我们的重点将是增强汽车在不完美的条件下正确地“看到”的能力，例如夜间驾驶、下雨驾驶或雪地驾驶。这项技术也可能对任何与视力问题作斗争的人有用，包括老年人。可以通过现有的可穿戴显示器、清晰的显示器、通过投影到挡风玻璃上等来提供增强的图像。任何视力受限的人都能从中受益良多。

This research program proposes to develop new technology that will lead to improved object detection and computer vision for autonomous vehicles using machine learning. The technology will include both software and hardware for compute-acceleration using field-programmable gate arrays (FPGAs). FPGAs are a type of integrated circuit chip that can be programmed to implement different applications in hardware. The program comprises three related research thrusts: neural-network (NN) algorithms that incorporate sensor-fusion techniques, NN circuit architectures for FPGAs, and end-user applications of this technology, such as enhancing vision for auto-drivers in conditions that are difficult for those drivers to manage. The traditional approach to object detection for autonomous vehicles has been a modified image classification task, which locates objects (e.g. cars, pedestrians) in a frame as well as performing an object classification task. However, as research has progressed, promising results with Light Detection and Ranging (LiDAR) data has catalyzed research into fusion of LiDAR and vision (camera) sensors. In this research program we propose to explore and fully understand fusion operations like the element-wise mean, but taking into consideration the predicted performance of individual sensors in novel ways. The performance evaluation of sensors becomes useful when one sensor is compromised through weather, component breakdown or intrusion attempts. When a sensor is incorrectly making predictions, the fusion-prediction network must be able to dynamically compensate to avoid degradation of the entire object detection system. State-of-the-art computer vision solutions for autonomous vehicles increasing rely on convolutional neural networks (CNNs) to achieve good quality of result. A CNN typically has millions of neurons and synapses which incur high computational complexity and storage requirements. Therefore, deploying CNNs on autonomous cars and drones as object detectors and sensor fusion solutions is difficult because of the tight power budget, low latency requirement, and scarce computation resources in these platforms. FPGAs have emerged as a popular substrate for implementing dedicated CNN accelerators for these applications. Results from this research program will be applied to a multi-university international self-driving automotive competition being sponsored by General Motors. Our focus will be on enhancing an automobile's ability to "see" properly in imperfect conditions, such as night driving, rain driving or snow driving. This technology could also be useful for any person who struggles with vision issues, including the elderly. Enhanced images could be provided through an existing wearable display, a clear display, via projection onto a windshield, and so on. Any person who suffers from vision restrictions could benefit greatly.