CPS: Medium: Enabling Multimodal Sensing, Real-time Onboard Detection and Adaptive Control for Fully Autonomous Unmanned Aerial Systems

该项目的目标是研究一种低成本、高能效的硬件和软件系统，以关闭传感器数据处理、语义高层检测和实时轨迹生成之间的循环。为了将无人机安全地纳入国家领空，迫切需要发展机载感知和规避能力。虽然深度神经网络(DNN)显著提高了目标检测和决策的准确性，但它们在小型无人机上实现的复杂性高得令人望而却步。此外，现有的无人机飞行控制方法忽略了无人机的非线性，不能提供轨迹保证。本项目的研究重点是：(I)DNN的FPGA实现：深度(卷积)神经网络的全连通层和卷积层都将使用(分块)循环矩阵进行训练，并使用定制设计的通用快速傅立叶变换核在FPGA上实现。这项研究将使DNN的高效实现成为可能，将内存和计算复杂性分别从O(N2)降低到O(N)和O(NlogN)；(Ii)车载感知和回避的自主检测和感知：现有的区域检测神经网络将扩展到处理从不同角度拍摄的图像和多模式传感器输入；(Iii)实时路点和轨迹生成-将开发一种集成的轨迹生成和反馈控制方案，用于在3D空间中引导欠驱动车辆通过所需的路点。为了有效地实现和硬件重用，检测和控制问题都将使用具有(块)循环权重矩阵的DNN来描述和解决。将研究深度强化学习模型以生成路点，并在障碍物周围分配人工势以保证安全距离。基础研究成果将实现机载计算、实时检测和控制，这是自主和下一代无人机的基石。

The goal of this project is to investigate a low-cost and energy-efficient hardware and software system to close the loop between processing of sensor data, semantically high-level detection and trajectory generation in real-time. To safely integrate Unmanned Aerial Vehicles into national airspace, there is an urgent need to develop onboard sense-and-avoid capability. While deep neural networks (DNNs) have significantly improved the accuracy of object detection and decision making, they have prohibitively high complexity to be implemented on small UAVs. Moreover, existing UAV flight control approaches ignore the nonlinearities of UAVs and do not provide trajectory assurance. The research thrusts of this project are: (i) FPGA implementation of DNNs: both fully connected and convolutional layers of deep (convolutional) neural networks will be trained using (block-)circulant matrix and implemented using custom designed universal Fast Fourier Transform kernels on FPGA. This research thrust will enable efficient implementation of DNNs, reducing memory and computation complexity from O(N2) to O(N) and O(NlogN), respectively; (ii) autonomous detection and perception for onboard sense-and-avoid: existing regional detection neural networks will be extended to work with images taken from different angles, and multi-modal sensor inputs; (iii) real-time waypoint and trajectory generation - an integrated trajectory generation and feedback control scheme for steering under-actuated vehicles through desired waypoints in 3D space will be developed. For efficient implementation and hardware reuse, both detection and control problems will be formulated and solved using DNNs with (block-)circulant weight matrix. Deep reinforcement learning models will be investigated for waypoint generation and to assign artificial potential around the obstacles to guarantee a safe distance. The fundamental research results will enable onboard computing, real-time detection and control, which are cornerstones of autonomous and next-generation UAVs.