CAREER: Heterogeneous Neuromorphic and Edge Computing Systems for Realtime Machine Learning Technologies

机器人、自动驾驶汽车、辅助技术和物联网(IoT)应用中的机器学习系统需要低能耗的实时计算。较低的能耗可确保延长这些电池供电设备的电池寿命。这个项目的重点是美国手语翻译，以展示其社会影响。为了创造实用的手语翻译技术，多种计算机视觉和语言模型对于手语使用者和其他人之间的无缝交流是必不可少的。其目标是将其部署在便携、可穿戴设备上，以按需使用-这是一个复杂的挑战。研究小组将研究分解这些复杂的系统，将计算分布在专门从事特定任务的相互连接的微型设备上。这项工作的成果可以增强听力和言语障碍者的能力，促进包容性沟通和劳动力多样性。除了手语翻译，该项目开发的方法和框架可以为社交机器人和智能制造等领域的实时技术铺平道路。该项目涉及各种教育和推广活动，包括开发跨学科课程，生成在线教育资源，让本科生和高中生参与研究，以及与行业合作伙伴合作，促进K-5学习的社会机器人。该项目旨在利用神经形态和边缘计算的组合能力来打造一个不同的机器学习系统。它的主要目标是在资源和能源有限的设备上以前所未有的规模实现计算机视觉和语言模型。它集中在几个关键方面：(1)开发将尖峰神经网络的能效、时间稀疏性和时空处理与复杂大规模计算机视觉任务的变压器模型的全局处理相结合的混合模型；(2)通过采用系统级的创新，如计算图形修改、定制内核和数学重构，创建在边缘设备上部署大型语言模型的方法；(3)设计灵活的边缘人工智能(AI)加速器，以克服阻碍在边缘实时实施大型变压器模型的硬件限制；(4)无缝集成移动处理器、边缘AI加速器、和神经形态硬件，提供全面的端到端解决方案。在整个项目中，严谨的调查深入研究带宽、准确性、性能和能源消耗之间的关键权衡。该项目由软件和硬件基金会(SHF)核心研究计划和既定的激励竞争研究计划(EPSCoR)联合资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning systems in robotics, self-driving cars, assistive technologies, and Internet-of-Things (IoT) applications require low-energy, real-time computation. Lower energy use ensures extended battery life for these battery-powered devices. This project focuses on American sign language translation to showcase its societal impact. To create practical sign language translation technology, multiple computer vision and language models are essential for seamless communication between sign language users and others. The aim is to deploy this on portable, wearable devices for on-demand use - a complex challenge. The research team will investigate breaking down these complex systems, distributing computation across interconnected tiny devices specialized in specific tasks. The outcome of this work can empower those with hearing and speech impairments, fostering inclusive communication and workforce diversity. Beyond sign language translation, the methodology and framework developed in this project can pave the way for real-time technology in social robotics and smart manufacturing, among other domains. This project involves various educational and outreach initiatives, including developing cross-disciplinary curricula, generating online educational resources, engaging both undergraduate and high school students in research, and collaborating with industry partners to promote social robotics for K-5 learning.This project aims to harness the combined capabilities of neuromorphic and edge computing to forge a heterogeneous machine learning system. Its primary goal is to enable computer vision and language models on resource- and energy-constrained devices at an unprecedented scale. It focuses on several key aspects: (1) developing hybrid models that merge the energy efficiency, temporal sparsity, and spatiotemporal processing of spiking neural networks with the global processing of transformer models for complex large-scale computer vision tasks, (2) creating a methodology to deploy large language models on edge devices by employing system-level innovations such as computational graph modifications, custom kernels, and mathematical refactoring, (3) designing a flexible edge artificial intelligence (AI) accelerator to overcome hardware limitations hindering real-time implementation of large transformer models at the edge, (4) seamlessly integrating a heterogeneous system of mobile processors, edge AI accelerators, and neuromorphic hardware for a comprehensive end-to-end solution. Throughout the project, rigorous investigation delves into critical trade-offs between bandwidth, accuracy, performance, and energy consumption.This project is jointly funded by the Software and Hardware Foundation (SHF) core research program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.