Collaborative Research: SHF: Medium: Neural-Network-based Stochastic Computing Architectures with applications to Machine Learning

现代计算硬件受到诸如极小尺寸、低功耗和高可靠性的严格要求的约束。因此，直接解决这些问题的非常规计算方法（例如随机计算（SC））越来越受到关注，特别是对于人工智能（AI）中的机器学习（ML）应用。SC是一种新颖的计算框架，其中输入数据作为比特流连续提供;因此，复杂的计算可以通过对流进行简单的逐位操作来计算。SC的主要吸引力在于它能够实现非常低成本和低功耗的架构实现，特别是对于使用简单逻辑元件的算术运算。这个特性与神经网络（NN）非常相关，因为NN需要大量的硬件资源，因此在处理ML的大数据集时会消耗大量的功率。 此外，目前的神经网络架构很难配置，以适应不同的应用，因为硬件是相当复杂的，不是很灵活。因此，随着ML系统使用NN达到计算的基本极限，SC已经成为一种合理而实用的解决方案，可以满足大规模并行和快速部署硬件的性能，能源和弹性要求，以支持AI，对技术和国民经济增长产生直接影响。该项目的目标是开发NN架构，该架构依赖于不同的计算功能，用于设计这种高效的基于SC的NN的横切方案（跨越硬件单元，算法和应用程序）。该项目下的技术工作利用了SC的主要功能，并提出了一个合理的研究计划，具有几个新的概念。这项研究的第一个新奇在于，它使SC NN的设计成为可能，因为它专注于架构级硬件目标，也是SC的重要指标（例如减少延迟和提高准确性，主要是在推理和训练中）。这项工作的第二个新奇是，它解决了基本问题，其中简单的SC硬件是利用自适应的数据，以维持高水平的并行计算在NN;解决方案围绕一个可配置的自下而上的计划，其中最初的低级别的硬件（如神经元和处理功能单元）模块化采用在NN，以支持在更高级别的计算。新的内存组织，以弥补错误时，SC也提出了，这也提高了应用程序相关的要求。第三个新奇是在一个组合的硬件实现上提供SC以及常规（二进制）计算;这是在SC不满足手头应用的精度要求的情况下优化计算性能的额外益处。因此，这项及时的研究旨在为新兴的计算系统和架构提供持续的技术创新，这些系统和架构与计算和ML社区都相关，并对社会和美国计算行业的进步产生重大影响。此外，本发明还该项目致力于扩大参与计算（BPC）该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computing hardware is constrained by stringent requirements such as extremely small size, low power consumption, and high reliability. Consequently, unconventional computing methods, such as Stochastic Computing (SC), that directly address these issues are of increasing interest, especially for Machine Learning (ML) applications in Artificial Intelligence (AI). SC is a novel computation framework in which input data is continuously provided as a streams of bits; therefore, complex computations can then be computed by simple bit-wise operations on the streams. The main attraction of SC is that it enables very low-cost and low-power architectural implementations, especially for arithmetic operations using simple logic elements. This feature is very relevant to Neural Networks (NNs), because NNs require significant hardware resources, therefore consuming substantial power when processing big datasets for ML. Moreover, current NN architectures are difficult to configure to suit different applications, because the hardware is rather complex and not very flexible. Thus, as ML systems are reaching the fundamental limits of computation using NNs, SC has emerged as a plausible and practical solution to meet performance, energy and resilience requirements for massive parallelism and fast deployment of hardware to support AI with direct impact on technology and national economic growth. The goal of this project is to develop NN architectures that rely on different computational features for cross-cutting schemes (spanning hardware units, algorithms, and applications) aimed at designing such efficient SC-based NNs.The technical work pursued under this project exploits the main features of SC and proposes a sound research program with several novel concepts. The first novelty of this investigation is that it makes possible the design of SC NNs by focusing on architectural-level hardware targeting also important metrics for SC (such as reducing latency and improving accuracy, mostly in inference and training). The second novelty of this work is that it addresses fundamental issues in which simple SC hardware is utilized adaptively to data to sustain a high level of parallel computation in NNs; solutions revolve around a configurable bottom-up scheme in which initially low-level hardware (such as neurons and processing function units) are modularly employed in the NNs to support computation at higher levels. Novel memory organizations to remedy errors when SC is employed are also proposed; this also enhances application-dependent requirements. The third novelty is the provision of having both SC as well as conventional (binary) computation on one combined hardware implementation; this is an added benefit for optimizing computing performance just in case the SC does not meet the accuracy requirements of the application at hand. Therefore, this timely research is directed to the continued technical innovation for emerging computing systems and architectures with relevance to both the computing and ML communities and strong implications on advancements in society and the US computing industry-at-large; moreover, this project is strongly committed to Broadening Participation in Computing (BPC) and its success.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.