SHF: Small: Collaborative Research: LDPD-Net: A Framework for Accelerated Architectures for Low-Density Permuted-Diagonal Deep Neural Networks

深度学习已经成为机器学习的一种重要形式，多层神经网络可以从可用的输入输出数据中学习系统功能。深度学习在图像识别、医疗保健和自动驾驶汽车等领域的表现优于基于特征工程的传统机器学习算法。它们在云计算中被广泛使用，在云计算中有大量的计算资源可用。深度神经网络通常使用图形处理单元(GPU)或张量处理单元(TPU)进行训练。随着神经网络复杂度的增加，训练时间和能量消耗也随之增加。该项目试图将稀疏性和规则性作为对深层神经网络结构的约束，以降低复杂性和能量消耗数量级，可能会以性能略有下降为代价。其影响在于形成了一种新的神经网络结构家族，称为低密度置换对角网络或LDPD-Net。该方法将使深度神经网络能够部署在能源受限和资源受限的嵌入式平台上，用于推理任务，包括但不限于无人机/航空系统、个性化医疗保健、可穿戴和可植入设备以及移动智能系统。此外，本项目开发的设计方法/技术可以促进对其他基于矩阵/张量的大数据处理和分析方法的有效计算的研究。这些方法还可能在数据驱动的神经科学和数据驱动的信号处理中得到应用。除了研究生，该项目还将通过高级设计项目和本科生的研究经验吸引本科生。该项目的成果将通过出版物、演讲、在不同行业和其他学术机构的讲座向更广泛的社区传播。深度学习网络广泛应用的主要障碍包括计算资源限制和能量消耗限制。这些障碍可以通过在深度神经网络的不同层之间施加稀疏性和规则性来放松。所提出的低密度置换对角线(LDPD)网络可以在计算复杂度、存储空间和能量消耗方面降低数量级。LDPD-Net不会通过首先训练规则网络，然后只保留与LDPD-Net对应的权重来重新训练。相反，拟议中的网络将从头开始培训。提出的LDPD-Net能够针对特定的计算平台实现网络的伸缩。本文的研究内容包括三个方面：1)开发新的资源受限和能量受限的推理和训练系统；2)开发能够充分利用LDPD网的优势以获得高性能的新型高效硬件结构；3)进行新颖的软硬件协同设计和协同优化，以探索LDPD网的设计空间。利用这些，将通过在高性能系统、低功耗嵌入式系统上的软件实施以及在FPGA开发板上的硬件原型来验证和评估建议的LDPD-Net的有效性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has emerged as an important form of machine-learning where multiple layers of neural networks can learn the system function from available input-output data. Deep learning has outperformed traditional machine-learning algorithms based on feature engineering in fields such as image recognition, healthcare, and autonomous vehicles. These are widely used in cloud computing where large amount of computational resources are available. Deep neural networks are typically trained using graphic processing units (GPUs) or tensor processing units (TPUs). The training time and energy consumption grow with the complexity of the neural network. This project attempts to impose sparsity and regularity as constraints on the structure of the deep neural networks to reduce complexity and energy consumption by orders of magnitude, possibly at the expense of a slight degradation in the performance. The impacts lie in the formulation of a new family of structures for neural networks referred to as Low-Density Permuted Diagonal Network or LDPD-Net. The approach will enable the deployment of deep neural networks in energy-constrained and resource-constrained embedded platforms for inference tasks, including, but not limited to, unmanned vehicles/aerial systems, personalized healthcare, wearable and implantable devices, and mobile intelligent systems. In addition, the design methodology/techniques developed in this project can facilitate investigation of efficient computing of other matrix/tensor-based big data processing and analysis approaches. These approaches may also find applications in data-driven neuroscience and data-driven signal processing. In addition to graduate students, the project will involve undergraduates via senior design projects and research experiences for undergraduates. The results of the project will be disseminated to the broader community by publications, presentations, talks at various industries and other academic institutions. The main barriers to wide adoption of deep learning networks include computational resource constraints and energy consumption constraints. These barriers can be relaxed by imposing sparsity and regularity among different layers of the deep neural network. The proposed low-density permuted-diagonal (LDPD) network can lead to orders of magnitude reduction in computation complexity, storage space and energy consumption. The LDPD-Net will not be retrained by first training a regular network and then only retaining the weights corresponding to the LDPD-Net. Instead, the proposed network will be trained from scratch. The proposed LDPD-Net can enable scaling of the network for a specified computational platform. The proposed research has three thrusts: 1) develop novel resource-constrained and energy-constrained inference and training systems; 2) develop novel efficient hardware architectures that can fully exploit the advantages of the LDPD-Net to achieve high performance; and 3) perform novel software and hardware co-design and co-optimization to explore the design space of the LDPD-Net. Using these, the efficacy of the proposed LDPD-net will be validated and evaluated, via software implementations on high-performance systems, low-power embedded systems, and a hardware prototype on FPGA development boards.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.