Collaborative Research: SHF: Small: Quasi Weightless Neural Networks for Energy-Efficient Machine Learning on the Edge

深度神经网络（dnn）最近在各种各样的任务中取得了革命性的进步，然而这些深度网络需要大量的内存和计算资源。对于边缘系统来说，这样的需求可能非常困难（甚至不切实际）。虽然dnn非常精确，但dnn消耗的能量比类似任务的生物神经活动高几个数量级。减少机器学习硬件的计算和能源需求是很重要的，这样边缘推理就可以成为一项低成本、低能耗的任务。无重力神经网络（WNNs）是一类独特的神经模型，其灵感来源于生物神经元的树突树对输入信号的处理。wnn不使用权重或乘法运算来确定它们的响应。相反，它们依赖于使用查询表实现的值查找。与基于乘法和加法的深度学习模型相比，该项目探索了更节能的小模型。从能源效率和低延迟的角度来看，无线网络是非常有前途的，我们的努力是为了实现无数的超低能量边缘应用，否则是不可能的。该项目探索了低能耗的机器学习硬件，它结合了传统深度神经网络和无计算权重神经网络的优点。所使用的技术包括：(1)利用多层网络和分层网络创建新的无权重神经网络架构；(2)利用多镜头反馈训练为无权重神经网络设计新的训练算法；(3)利用新兴的新型记忆技术探索准无权重神经网络；(4)设计节能边缘智能系统。德克萨斯大学和斯坦福大学之间的合作项目在系统堆栈的多个层面进行创新，包括架构和电路层。从STEM的角度来看，德克萨斯大学和斯坦福大学之间的合作活动涉及许多代表性不足的社区，包括少数民族和女性、本科生和第一代大学生。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep Neural Networks (DNNs) have recently enabled revolutionary advances in a wide variety of tasks, however these deep networks demand large amounts of memory and computation resources. Such demands can be highly difficult (or even impractical) for systems on the edge. Although DNNs are very accurate, the energy consumed by DNNs is orders of magnitude higher than biological neural activities for similar tasks. It is important to reduce the computational and energy demands of machine learning hardware so that inferencing on the edge can become a low-cost, low-energy task. Weightless Neural Networks (WNNs) represent a distinct class of neural models which derive inspiration from the processing of input signals by the dendritic trees of biological neurons. WNNs do not use weights or multiply-add operations to determine their responses. Instead, they rely on value lookups implemented using look-up tables. This project explores small models that are more energy efficient compared to multiplication-and-addition-based deep learning models. WNNs are very promising from the perspective of energy-efficiency, and low latency, and our effort is directed at enabling a myriad of ultra-low energy edge applications otherwise impossible. This project explores low-energy machine learning hardware which combine the benefits of traditional DNNs and the computation-less weightless neural networks. Techniques used include (1) utilizing multi-layer networks and hierarchical networks to create novel weightless neural network architectures, (2) devising novel training algorithms for WNNs utilizing multi-shot training with feedback (3) exploring quasi-weightless neural networks using emerging novel memory technologies, and (4) designing systems for energy-efficient edge intelligence. The collaborative project between the University of Texas and Stanford University innovates across multiple layers of the system stack, including architecture and circuit layers. The collaborative activity between the University of Texas and Stanford involves many underrepresented communities from a STEM perspective, including minority and women, undergrads, and first-generation college students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.