SHF: Small: Collaborative Research: LDPD-Net: A Framework for Accelerated Architectures for Low-Density Permuted-Diagonal Deep Neural Networks

深度学习已经成为机器学习的一种重要形式，多层神经网络可以从可用的输入输出数据中学习系统函数。深度学习在图像识别、医疗保健和自动驾驶汽车等领域的表现优于基于特征工程的传统机器学习算法。这些被广泛用于云计算，其中大量的计算资源是可用的。深度神经网络通常使用图形处理单元（GPU）或张量处理单元（TPU）进行训练。训练时间和能量消耗随着神经网络的复杂性而增长。该项目试图将稀疏性和规则性作为对深度神经网络结构的约束，以降低复杂性和能量消耗的数量级，可能以性能轻微下降为代价。其影响在于神经网络的一种新结构家族的形成，称为低密度置换对角网络或LDPD-Net。该方法将使深度神经网络能够在能量受限和资源受限的嵌入式平台中部署，用于推理任务，包括但不限于无人驾驶车辆/航空系统，个性化医疗保健，可穿戴和植入式设备以及移动的智能系统。此外，本项目开发的设计方法/技术可以促进对其他基于矩阵/张量的大数据处理和分析方法的有效计算的研究。这些方法也可以在数据驱动的神经科学和数据驱动的信号处理中找到应用。除了研究生外，该项目还将通过高级设计项目和本科生的研究经验来吸引本科生。该项目的成果将通过出版物、介绍会、在各行业和其他学术机构的讲座等方式向更广泛的社区传播。广泛采用深度学习网络的主要障碍包括计算资源约束和能耗约束。这些障碍可以通过在深度神经网络的不同层之间施加稀疏性和规则性来放松。所提出的低密度置换对角（LDPD）网络可以导致计算复杂度，存储空间和能量消耗的数量级降低。LDPD-Net不会通过首先训练常规网络然后仅保留与LDPD-Net对应的权重来重新训练。相反，拟议的网络将从头开始训练。所提出的LDPD-Net可以针对指定的计算平台实现网络的缩放。该研究有三个主要目标：1）开发新的资源受限和能量受限的推理和训练系统; 2）开发新的高效硬件架构，可以充分利用LDPD-Net的优势，以实现高性能; 3）执行新的软件和硬件协同设计和协同优化，以探索LDPD-Net的设计空间。通过在高性能系统、低功耗嵌入式系统和FPGA开发板上的硬件原型上的软件实现，验证和评估拟议中的LDPD-net的有效性。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has emerged as an important form of machine-learning where multiple layers of neural networks can learn the system function from available input-output data. Deep learning has outperformed traditional machine-learning algorithms based on feature engineering in fields such as image recognition, healthcare, and autonomous vehicles. These are widely used in cloud computing where large amount of computational resources are available. Deep neural networks are typically trained using graphic processing units (GPUs) or tensor processing units (TPUs). The training time and energy consumption grow with the complexity of the neural network. This project attempts to impose sparsity and regularity as constraints on the structure of the deep neural networks to reduce complexity and energy consumption by orders of magnitude, possibly at the expense of a slight degradation in the performance. The impacts lie in the formulation of a new family of structures for neural networks referred to as Low-Density Permuted Diagonal Network or LDPD-Net. The approach will enable the deployment of deep neural networks in energy-constrained and resource-constrained embedded platforms for inference tasks, including, but not limited to, unmanned vehicles/aerial systems, personalized healthcare, wearable and implantable devices, and mobile intelligent systems. In addition, the design methodology/techniques developed in this project can facilitate investigation of efficient computing of other matrix/tensor-based big data processing and analysis approaches. These approaches may also find applications in data-driven neuroscience and data-driven signal processing. In addition to graduate students, the project will involve undergraduates via senior design projects and research experiences for undergraduates. The results of the project will be disseminated to the broader community by publications, presentations, talks at various industries and other academic institutions. The main barriers to wide adoption of deep learning networks include computational resource constraints and energy consumption constraints. These barriers can be relaxed by imposing sparsity and regularity among different layers of the deep neural network. The proposed low-density permuted-diagonal (LDPD) network can lead to orders of magnitude reduction in computation complexity, storage space and energy consumption. The LDPD-Net will not be retrained by first training a regular network and then only retaining the weights corresponding to the LDPD-Net. Instead, the proposed network will be trained from scratch. The proposed LDPD-Net can enable scaling of the network for a specified computational platform. The proposed research has three thrusts: 1) develop novel resource-constrained and energy-constrained inference and training systems; 2) develop novel efficient hardware architectures that can fully exploit the advantages of the LDPD-Net to achieve high performance; and 3) perform novel software and hardware co-design and co-optimization to explore the design space of the LDPD-Net. Using these, the efficacy of the proposed LDPD-net will be validated and evaluated, via software implementations on high-performance systems, low-power embedded systems, and a hardware prototype on FPGA development boards.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.