SHF: Medium: Collaborative Research: ADMM-NN: A Unified Software/Hardware Framework of DNN Computation and Storage Reduction Using ADMM

深度神经网络（DNN）由于其卓越的性能而被广泛应用于各种应用领域。DNN的硬件实现对于无处不在的嵌入式和物联网（IoT）设备至关重要，这些设备要求在能源和资源受限的系统中实现高性能。该项目旨在解决将复杂的DNN模型映射到硬件以实现节能和性能驱动的实现时所面临的挑战。所提出的技术将促进深度学习在高性能和低功耗计算系统中的广泛采用。该项目还将通过支持更广泛采用智能系统进行大数据分析、天气建模和预测、疾病诊断和药物输送以及医学图像处理的解决方案，增加经济机会并产生重大的社会效益。研究进展将纳入调查人员教授的课程。参与代表性不足，本科生和K12学生的活动将与东北大学STEM教育中心和南加州大学维特比工程多样性中心合作设计。该项目的所有软件代码将通过GitHub发布，教育模块和教程将提供给研究社区，行业和政府。探索DNN的固有模型冗余，该项目将开发一个算法-硬件协同优化框架，通过利用ADMM（交替方向乘法器方法），一种强大的优化技术，大大减少DNN的计算和存储需求。该项目首先解决了由于DNN训练中的非凸目标函数而导致的ADMM应用中的挑战，从而缺乏对解决方案可行性，解决方案质量和低运行时间的保证。因此，ADMM正则化和掩码映射和再训练的集成框架将被开发，并将研究解决方案质量，性能驱动的计算/存储减少和硬件可行性的进一步改进。接下来，该项目提出了一种统一的权重和中间结果修剪和量化技术，探索DNN模型的所有四个冗余源。由于DNN的硬件实现对能源效率的影响，几乎所有的DNN模型，或者至少是计算最密集的卷积层，都可以放在单个芯片上。最后，将开发设计时参数化和算法-硬件协同设计解决方案，以有效利用可用硬件资源，实现高性能、高能效和自适应能力。将进行广泛的实验和评估，以验证和调整所提出的技术与原型系统使用FPGA设备。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

Deep neural networks (DNNs) have been employed in wide application domains thanks to their extraordinary performance. Hardware implementations of DNNs are of critical importance for the ubiquitous embedded and Internet of Things (IoT) devices, which call for high performance in energy and resource constrained systems. This project aims to address the challenges when mapping complicated DNN models into hardware for energy-efficient and performance-driven implementations. The proposed techniques will promote wider adoptions of deep learning into both high-performance and low-power computing systems. The project will also enhance economic opportunities and have significant societal benefits via solutions that support broader adoption of intelligent systems for big data analytics, weather modeling and forecasting, disease diagnosis and drug delivery, and medical image processing. The research advances will be incorporated into coursework taught by the investigators. Activities on engaging underrepresented, undergraduate, and K12 students will be designed in collaboration with the Northeastern University Center of STEM Education and University of Southern California's Viterbi Center for Engineering Diversity. All software code from the project will be released via GitHub and educational modules and tutorials will be make available to the research community, industry, and government. Exploring the inherent model redundancy of DNNs, this project will develop an algorithm-hardware co-optimization framework for greatly reducing DNN computation and storage requirements by leveraging ADMM (alternating direction method of multipliers), a powerful optimization technique. This project first solves the challenge in the application of ADMM due to the non-convex objective function in DNN training, and thereby lack of guarantees on solution feasibility, solution quality, and low runtime. Therefore, an integrated framework of ADMM regularization and masked mapping and retraining will be developed and further improvements on solution quality, performance-driven computation/storage reduction, and hardware feasibility will be investigated. Next, the project proposes a unified weight and intermediate result pruning and quantization technique that explores all four redundancy sources of DNN models. Due to the impact on energy efficiency of hardware implementations of DNNs, nearly all DNN models, or at least the most computationally intensive convolutional layers can be then placed on a single chip. Finally, design-time parameterization and algorithm-hardware co-design solutions will be developed for efficient utilization of available hardware resources, achieving high performance, energy efficiency, and adaptation capability. Extensive experimentation and evaluation will be performed to validate and tune the proposed technique with prototype systems using FPGA devices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.