SHF: Medium: Collaborative Research: ADMM-NN: A Unified Software/Hardware Framework of DNN Computation and Storage Reduction Using ADMM

SHF：中：协作研究：ADMM-NN：使用 ADMM 进行 DNN 计算和存储缩减的统一软硬件框架

• 批准号: 1901440
• 负责人: Massoud Pedram
• 金额: $ 45万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-06-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1901440&HistoricalAwards=false
• 关键词: SHF; Medium; Collaborative; Research; ADMM

Deep neural networks (DNNs) have been employed in wide application domains thanks to their extraordinary performance. Hardware implementations of DNNs are of critical importance for the ubiquitous embedded and Internet of Things (IoT) devices, which call for high performance in energy and resource constrained systems. This project aims to address the challenges when mapping complicated DNN models into hardware for energy-efficient and performance-driven implementations. The proposed techniques will promote wider adoptions of deep learning into both high-performance and low-power computing systems. The project will also enhance economic opportunities and have significant societal benefits via solutions that support broader adoption of intelligent systems for big data analytics, weather modeling and forecasting, disease diagnosis and drug delivery, and medical image processing. The research advances will be incorporated into coursework taught by the investigators. Activities on engaging underrepresented, undergraduate, and K12 students will be designed in collaboration with the Northeastern University Center of STEM Education and University of Southern California's Viterbi Center for Engineering Diversity. All software code from the project will be released via GitHub and educational modules and tutorials will be make available to the research community, industry, and government. Exploring the inherent model redundancy of DNNs, this project will develop an algorithm-hardware co-optimization framework for greatly reducing DNN computation and storage requirements by leveraging ADMM (alternating direction method of multipliers), a powerful optimization technique. This project first solves the challenge in the application of ADMM due to the non-convex objective function in DNN training, and thereby lack of guarantees on solution feasibility, solution quality, and low runtime. Therefore, an integrated framework of ADMM regularization and masked mapping and retraining will be developed and further improvements on solution quality, performance-driven computation/storage reduction, and hardware feasibility will be investigated. Next, the project proposes a unified weight and intermediate result pruning and quantization technique that explores all four redundancy sources of DNN models. Due to the impact on energy efficiency of hardware implementations of DNNs, nearly all DNN models, or at least the most computationally intensive convolutional layers can be then placed on a single chip. Finally, design-time parameterization and algorithm-hardware co-design solutions will be developed for efficient utilization of available hardware resources, achieving high performance, energy efficiency, and adaptation capability. Extensive experimentation and evaluation will be performed to validate and tune the proposed technique with prototype systems using FPGA devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度神经网络（DNN）由于其卓越的性能而被广泛应用于各种应用领域。DNN的硬件实现对于无处不在的嵌入式和物联网（IoT）设备至关重要，这些设备要求在能源和资源受限的系统中实现高性能。该项目旨在解决将复杂的DNN模型映射到硬件以实现节能和性能驱动的实现时所面临的挑战。所提出的技术将促进深度学习在高性能和低功耗计算系统中的广泛采用。该项目还将通过支持更广泛采用智能系统进行大数据分析、天气建模和预测、疾病诊断和药物输送以及医学图像处理的解决方案，增加经济机会并产生重大的社会效益。研究进展将纳入调查人员教授的课程。参与代表性不足，本科生和K12学生的活动将与东北大学STEM教育中心和南加州大学维特比工程多样性中心合作设计。该项目的所有软件代码将通过GitHub发布，教育模块和教程将提供给研究社区，行业和政府。探索DNN的固有模型冗余，该项目将开发一个算法-硬件协同优化框架，通过利用ADMM（交替方向乘法器方法），一种强大的优化技术，大大减少DNN的计算和存储需求。该项目首先解决了由于DNN训练中的非凸目标函数而导致的ADMM应用中的挑战，从而缺乏对解决方案可行性，解决方案质量和低运行时间的保证。因此，ADMM正则化和掩码映射和再训练的集成框架将被开发，并将研究解决方案质量，性能驱动的计算/存储减少和硬件可行性的进一步改进。接下来，该项目提出了一种统一的权重和中间结果修剪和量化技术，探索DNN模型的所有四个冗余源。由于DNN的硬件实现对能源效率的影响，几乎所有的DNN模型，或者至少是计算最密集的卷积层，都可以放在单个芯片上。最后，将开发设计时参数化和算法-硬件协同设计解决方案，以有效利用可用硬件资源，实现高性能、高能效和自适应能力。将进行广泛的实验和评估，以验证和调整所提出的技术与原型系统使用FPGA设备。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

项目成果

DNR: A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs

• DOI: 10.1145/3394885.3431542
• 发表时间: 2020-11
• 期刊: 2021 26th Asia and South Pacific Design Automation Conference (ASP-DAC)
• 作者: Souvik Kundu;M. Nazemi;P. Beerel;M. Pedram

Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators

• DOI: 10.1145/3531437.3539715
• 发表时间: 2022-06
• 期刊: Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design
• 作者: J. Heo;A. Fayyazi;Amirhossein Esmaili;M. Pedram

An Energy-Efficient Inference Method in Convolutional Neural Networks Based on Dynamic Adjustment of the Pruning Level

一种基于动态调整剪枝水平的卷积神经网络节能推理方法

• DOI: 10.1145/3460972
• 发表时间: 2021
• 期刊: ACM Transactions on Design Automation of Electronic Systems
• 影响因子: 1.4
• 作者: Maleki, Mohammad-Ali;Nabipour-Meybodi, Alireza;Kamal, Mehdi;Afzali-Kusha, Ali;Pedram, Massoud
• 通讯作者: Pedram, Massoud

SynergicLearning: neural network-based feature extraction for highly-accurate hyperdimensional learning

SynergicLearning：基于神经网络的特征提取，用于高精度超维学习

• DOI: 10.1145/3400302.3415696
• 发表时间: 2020
• 期刊: Int’l Conf. on Computer-Aided Design
• 作者: Nazemi, Mahdi;Fayyazi, Arash;Esmaili, Amirhossein;Pedram, Massoud
• 通讯作者: Pedram, Massoud

JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services

• DOI: 10.1109/tmc.2019.2947893
• 发表时间: 2021-02-01
• 期刊: IEEE TRANSACTIONS ON MOBILE COMPUTING
• 影响因子: 7.9
• 作者: Eshratifar, Amir Erfan;Abrishami, Mohammad Saeed;Pedram, Massoud
• 通讯作者: Pedram, Massoud