Collaborative Research: SHF: Small: Quasi Weightless Neural Networks for Energy-Efficient Machine Learning on the Edge

深度神经网络(DNN)最近在各种任务中取得了革命性的进展，但这些深层网络需要大量的存储和计算资源。对于处于边缘的系统来说，这样的要求可能非常困难(甚至不切实际)。虽然DNN非常准确，但对于类似的任务，DNN消耗的能量比生物神经活动高出几个数量级。重要的是要减少机器学习硬件的计算和能源需求，这样边缘推理就可以成为一项低成本、低能源的任务。失重神经网络(WNN)代表了一类不同的神经模型，其灵感来源于生物神经元的树突树对输入信号的处理。WNN不使用权重或乘加运算来确定其响应。相反，它们依赖于使用查找表实现的值查找。这个项目探索了比基于乘法和加法的深度学习模型更节能的小型模型。从能源效率和低延迟的角度来看，无线网络非常有前途，我们的努力旨在实现无数的超低能量边缘应用，否则是不可能的。这个项目探索了低能量机器学习硬件，它结合了传统DNN和无需计算的无重量神经网络的优点。所使用的技术包括(1)利用多层网络和分层网络来创建新的无重量神经网络结构，(2)利用带反馈的多次训练来设计新的WNN训练算法，(3)使用新兴的新型记忆技术来探索准无重量神经网络，以及(4)设计能量高效的边缘智能系统。德克萨斯大学和斯坦福大学的合作项目跨系统堆栈的多个层进行创新，包括体系结构层和电路层。德克萨斯大学和斯坦福大学之间的合作活动从STEM的角度涉及许多代表性不足的社区，包括少数族裔和女性、本科生和第一代大学生。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep Neural Networks (DNNs) have recently enabled revolutionary advances in a wide variety of tasks, however these deep networks demand large amounts of memory and computation resources. Such demands can be highly difficult (or even impractical) for systems on the edge. Although DNNs are very accurate, the energy consumed by DNNs is orders of magnitude higher than biological neural activities for similar tasks. It is important to reduce the computational and energy demands of machine learning hardware so that inferencing on the edge can become a low-cost, low-energy task. Weightless Neural Networks (WNNs) represent a distinct class of neural models which derive inspiration from the processing of input signals by the dendritic trees of biological neurons. WNNs do not use weights or multiply-add operations to determine their responses. Instead, they rely on value lookups implemented using look-up tables. This project explores small models that are more energy efficient compared to multiplication-and-addition-based deep learning models. WNNs are very promising from the perspective of energy-efficiency, and low latency, and our effort is directed at enabling a myriad of ultra-low energy edge applications otherwise impossible. This project explores low-energy machine learning hardware which combine the benefits of traditional DNNs and the computation-less weightless neural networks. Techniques used include (1) utilizing multi-layer networks and hierarchical networks to create novel weightless neural network architectures, (2) devising novel training algorithms for WNNs utilizing multi-shot training with feedback (3) exploring quasi-weightless neural networks using emerging novel memory technologies, and (4) designing systems for energy-efficient edge intelligence. The collaborative project between the University of Texas and Stanford University innovates across multiple layers of the system stack, including architecture and circuit layers. The collaborative activity between the University of Texas and Stanford involves many underrepresented communities from a STEM perspective, including minority and women, undergrads, and first-generation college students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.