NN-Thunder: HW/SW Codesign for Accelerating DNNs with Heterogeneous Beyond-von Neumann Architectures

深度学习和深度神经网络(DNN)已经在许多应用领域被采用，并具有重塑人类未来的巨大潜力。然而，最近的研究表明，当涉及深度学习时，经典的冯·诺伊曼体系结构天生效率低下，因为如此繁重的工作负载导致向以数据为中心的计算的非凡转变。在这种情况下，不可避免地要花费大量能量在处理元素和内存块之间来回移动大量数据。在这个项目中，我们设想了一个异质的底层硬件，它结合了von Neumann和Beyond-von Neumann架构的各种类型的加速器，在计算精度、消耗的能量、延迟和面积占用之间提供了广泛的折衷。我们的项目旨在解决Beyond-von Neumann建筑W.r.t.的局限性和改进。性能和能效。在Beyond-Von Neumann体系结构上缺乏对合适的神经网络的基础性探索，阻碍了充分开发此类加速器潜力的可能性。在这方面，二进制化神经网络(BNN)为超高效的硬件实现提供了可能性，并与由新兴的Beyond-CMOS器件构建的新型Beyond-von Neumann架构实现了出色的协同效应。本项目建议书计划探索不同的方法。为了充分发挥Beyond-von Neumann神经网络加速器的强大功能，我们致力于建模、设计和优化，以实现以下目标：(1)针对BNN的新兴技术和LIM(内存中逻辑)和PIM(内存中处理)的抽象进行跨层建模；(2)针对Beyond-von Neumann架构的硬件感知NN优化；以及(3)硬件/软件协同设计和优化。总而言之，实现异质硬件架构是未来深度学习的关键。它通过硬件/软件联合设计实现了混合精度神经网络的超高效执行，并有效地允许探索不同的新折衷方案。

Deep learning and deep neural networks (DNNs) have been adopted in many application areas and have a great potential to reshape the future of humankind. However, recent studies have demonstrated that classical von Neumann architectures are inherently inefficient when it comes to deep learning because such heavy workloads result in an extraordinary shift towards data-centric computing. Under such scenarios, a significant amount of energy is inevitably spent in moving a massive amount of data back and forth between processing elements and memory blocks. In this project, we envision a heterogeneous underlying HW, that combines various types of accelerators from both von Neumann and beyond-von Neumann architectures, offering a wide range of tradeoffs between computation accuracy, consumed energy, latency, and area footprint. Our project intends to address the limitation and improvement of beyond-von Neumann architectures w.r.t. performance and energy efficiency. The lack of fundamental exploration of suitable neural networks on beyond-von Neumann architectures hinders the possibility to exploit the full potential of such accelerators. In this regard, binarized neural networks (BNNs) offer the possibility of ultra-efficient hardware implementation and outstanding synergy with novel beyond-von Neumann architectures that are built from emerging beyond-CMOS devices. This project proposal plans to explore different means w.r.t. modeling, design, and optimization, to fully unleash the power of beyond-von Neumann neural network accelerators with the following goals: (1) cross-layer modeling for emerging technologies and abstractions of LiM (Logic-in-Memory) and PiM (Processing-in-Memory) for BNNs, (2) hardware-aware NN optimization for beyond-von Neumann architecture, and (3) HW/SW Codesign and Optimization. All in all, realizing a heterogeneous HW architecture is a key for future deep learning. It enables an ultra-efficient execution of hybrid-precision neural networks through HW/SW codesign and effectively allows the possibility to explore different novel tradeoffs.