NN-Thunder: HW/SW Codesign for Accelerating DNNs with Heterogeneous Beyond-von Neumann Architectures

深度学习和深度神经网络（DNN）已被应用于许多应用领域，并具有重塑人类未来的巨大潜力。然而，最近的研究表明，在深度学习方面，经典的冯·诺依曼架构本质上是低效的，因为如此繁重的工作负载导致了向以数据为中心的计算的巨大转变。在这种情况下，大量的能量不可避免地花费在处理元件和存储器块之间来回移动大量数据上。在这个项目中，我们设想了一个异构的底层硬件，它结合了来自冯·诺依曼和超越冯·诺依曼架构的各种类型的加速器，在计算精度、能耗、延迟和占地面积之间提供了广泛的权衡。我们的项目旨在解决的限制和改进的超越冯诺依曼架构w.r.t.性能和能源效率。缺乏对超越冯·诺依曼架构的合适神经网络的基本探索，阻碍了利用这种加速器的全部潜力的可能性。在这方面，二值化神经网络（BNN）提供了超高效硬件实现的可能性，以及与新兴超CMOS器件构建的新型超冯·诺依曼架构的出色协同作用。本项目建议书计划探索不同的方法，建模、设计和优化，充分释放超越冯·诺伊曼神经网络加速器的力量，实现以下目标：（1）新兴技术的跨层建模以及BNN的LiM（内存逻辑）和PiM（内存处理）的抽象，（2）超越冯·诺伊曼架构的硬件感知NN优化，以及（3）硬件/软件协同设计和优化。总而言之，实现异构硬件架构是未来深度学习的关键。它通过硬件/软件协同设计实现了混合精度神经网络的超高效执行，并有效地允许探索不同的新权衡。

Deep learning and deep neural networks (DNNs) have been adopted in many application areas and have a great potential to reshape the future of humankind. However, recent studies have demonstrated that classical von Neumann architectures are inherently inefficient when it comes to deep learning because such heavy workloads result in an extraordinary shift towards data-centric computing. Under such scenarios, a significant amount of energy is inevitably spent in moving a massive amount of data back and forth between processing elements and memory blocks. In this project, we envision a heterogeneous underlying HW, that combines various types of accelerators from both von Neumann and beyond-von Neumann architectures, offering a wide range of tradeoffs between computation accuracy, consumed energy, latency, and area footprint. Our project intends to address the limitation and improvement of beyond-von Neumann architectures w.r.t. performance and energy efficiency. The lack of fundamental exploration of suitable neural networks on beyond-von Neumann architectures hinders the possibility to exploit the full potential of such accelerators. In this regard, binarized neural networks (BNNs) offer the possibility of ultra-efficient hardware implementation and outstanding synergy with novel beyond-von Neumann architectures that are built from emerging beyond-CMOS devices. This project proposal plans to explore different means w.r.t. modeling, design, and optimization, to fully unleash the power of beyond-von Neumann neural network accelerators with the following goals: (1) cross-layer modeling for emerging technologies and abstractions of LiM (Logic-in-Memory) and PiM (Processing-in-Memory) for BNNs, (2) hardware-aware NN optimization for beyond-von Neumann architecture, and (3) HW/SW Codesign and Optimization. All in all, realizing a heterogeneous HW architecture is a key for future deep learning. It enables an ultra-efficient execution of hybrid-precision neural networks through HW/SW codesign and effectively allows the possibility to explore different novel tradeoffs.