SHF: Medium: Collaborative Research: ADMM-NN: A Unified Software/Hardware Framework of DNN Computation and Storage Reduction Using ADMM

深度神经网络(DNN)以其优异的性能被广泛应用于各个领域。DNN的硬件实现对于无处不在的嵌入式和物联网(IoT)设备至关重要，这些设备要求在能源和资源受限的系统中具有高性能。该项目旨在解决将复杂的DNN模型映射到硬件以实现节能和性能驱动的实施时所面临的挑战。建议的技术将促进深度学习在高性能和低功耗计算系统中的更广泛采用。该项目还将通过支持大数据分析、天气建模和预报、疾病诊断和药物输送以及医学图像处理的智能系统的更广泛采用的解决方案，增加经济机会并产生显著的社会效益。研究进展将被纳入研究人员教授的课程中。将与东北大学STEM教育中心和南加州大学维特比工程多样性中心合作，设计吸引代表不足的本科生和K12学生的活动。该项目的所有软件代码将通过GitHub发布，教育模块和教程将提供给研究社区、行业和政府。探索DNN固有的模型冗余，本项目将开发一个算法-硬件联合优化框架，利用强大的优化技术ADMM(交替方向乘子法)，大幅降低DNN的计算和存储需求。该项目首先解决了在DNN训练中由于目标函数的非凸性而导致的ADMM在应用中的挑战，从而在解的可行性、解的质量、运行时间等方面缺乏保证。因此，将开发ADMM正则化和掩码映射和再训练的集成框架，并将在解质量、性能驱动的计算/存储减少和硬件可行性方面进行进一步改进。接下来，该项目提出了一种统一的权重和中间结果剪枝和量化技术，该技术探索了DNN模型的所有四个冗余来源。由于DNN的硬件实现对能源效率的影响，几乎所有DNN模型，或者至少是计算最密集的卷积层可以被放置在单个芯片上。最后，将开发设计时参数化和算法-硬件协同设计解决方案，以有效利用现有硬件资源，实现高性能、能效和适配能力。将进行广泛的实验和评估，以验证和调整建议的技术与使用现场可编程门阵列器件的原型系统。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) have been employed in wide application domains thanks to their extraordinary performance. Hardware implementations of DNNs are of critical importance for the ubiquitous embedded and Internet of Things (IoT) devices, which call for high performance in energy and resource constrained systems. This project aims to address the challenges when mapping complicated DNN models into hardware for energy-efficient and performance-driven implementations. The proposed techniques will promote wider adoptions of deep learning into both high-performance and low-power computing systems. The project will also enhance economic opportunities and have significant societal benefits via solutions that support broader adoption of intelligent systems for big data analytics, weather modeling and forecasting, disease diagnosis and drug delivery, and medical image processing. The research advances will be incorporated into coursework taught by the investigators. Activities on engaging underrepresented, undergraduate, and K12 students will be designed in collaboration with the Northeastern University Center of STEM Education and University of Southern California's Viterbi Center for Engineering Diversity. All software code from the project will be released via GitHub and educational modules and tutorials will be make available to the research community, industry, and government. Exploring the inherent model redundancy of DNNs, this project will develop an algorithm-hardware co-optimization framework for greatly reducing DNN computation and storage requirements by leveraging ADMM (alternating direction method of multipliers), a powerful optimization technique. This project first solves the challenge in the application of ADMM due to the non-convex objective function in DNN training, and thereby lack of guarantees on solution feasibility, solution quality, and low runtime. Therefore, an integrated framework of ADMM regularization and masked mapping and retraining will be developed and further improvements on solution quality, performance-driven computation/storage reduction, and hardware feasibility will be investigated. Next, the project proposes a unified weight and intermediate result pruning and quantization technique that explores all four redundancy sources of DNN models. Due to the impact on energy efficiency of hardware implementations of DNNs, nearly all DNN models, or at least the most computationally intensive convolutional layers can be then placed on a single chip. Finally, design-time parameterization and algorithm-hardware co-design solutions will be developed for efficient utilization of available hardware resources, achieving high performance, energy efficiency, and adaptation capability. Extensive experimentation and evaluation will be performed to validate and tune the proposed technique with prototype systems using FPGA devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.