CAREER: Heterogeneous Neuromorphic and Edge Computing Systems for Realtime Machine Learning Technologies

机器人、自动驾驶汽车、辅助技术和物联网（IoT）应用中的机器学习系统需要低能耗、实时计算。更低的能耗确保了这些电池供电设备的电池寿命更长。这个项目的重点是美国手语翻译，以展示其社会影响。为了创造实用的手语翻译技术，多种计算机视觉和语言模型对于手语使用者和其他人之间的无缝沟通至关重要。其目标是将其部署到便携式、可穿戴设备上，以供按需使用——这是一个复杂的挑战。研究小组将研究分解这些复杂的系统，将计算分配到专门从事特定任务的相互连接的微型设备上。这项工作的成果可以增强听力和语言障碍人士的权能，促进包容性沟通和劳动力多样性。除了手语翻译之外，该项目开发的方法和框架可以为社交机器人和智能制造等领域的实时技术铺平道路。该项目涉及各种教育和推广活动，包括开发跨学科课程，生成在线教育资源，让本科生和高中生参与研究，以及与行业合作伙伴合作，促进社会机器人在K-5学习中的应用。该项目旨在利用神经形态和边缘计算的综合能力来构建异构机器学习系统。它的主要目标是以前所未有的规模在资源和能源受限的设备上实现计算机视觉和语言模型。它侧重于几个关键方面：(1)开发混合模型，将峰值神经网络的能源效率、时间稀疏性和时空处理与复杂的大规模计算机视觉任务的变压器模型的全局处理相结合；(2)通过采用计算图修改、自定义核和数学重构等系统级创新，创建一种在边缘设备上部署大型语言模型的方法；(3)设计灵活的边缘人工智能（AI）加速器，以克服阻碍在边缘实时实现大型变压器模型的硬件限制；(4)无缝集成移动处理器、边缘人工智能加速器和神经形态硬件的异构系统，以实现全面的端到端解决方案。在整个项目中，严格的调查深入研究了带宽、精度、性能和能耗之间的关键权衡。本项目由软件和硬件基金会（SHF）核心研究计划和促进竞争研究的既定计划（EPSCoR）共同资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning systems in robotics, self-driving cars, assistive technologies, and Internet-of-Things (IoT) applications require low-energy, real-time computation. Lower energy use ensures extended battery life for these battery-powered devices. This project focuses on American sign language translation to showcase its societal impact. To create practical sign language translation technology, multiple computer vision and language models are essential for seamless communication between sign language users and others. The aim is to deploy this on portable, wearable devices for on-demand use - a complex challenge. The research team will investigate breaking down these complex systems, distributing computation across interconnected tiny devices specialized in specific tasks. The outcome of this work can empower those with hearing and speech impairments, fostering inclusive communication and workforce diversity. Beyond sign language translation, the methodology and framework developed in this project can pave the way for real-time technology in social robotics and smart manufacturing, among other domains. This project involves various educational and outreach initiatives, including developing cross-disciplinary curricula, generating online educational resources, engaging both undergraduate and high school students in research, and collaborating with industry partners to promote social robotics for K-5 learning.This project aims to harness the combined capabilities of neuromorphic and edge computing to forge a heterogeneous machine learning system. Its primary goal is to enable computer vision and language models on resource- and energy-constrained devices at an unprecedented scale. It focuses on several key aspects: (1) developing hybrid models that merge the energy efficiency, temporal sparsity, and spatiotemporal processing of spiking neural networks with the global processing of transformer models for complex large-scale computer vision tasks, (2) creating a methodology to deploy large language models on edge devices by employing system-level innovations such as computational graph modifications, custom kernels, and mathematical refactoring, (3) designing a flexible edge artificial intelligence (AI) accelerator to overcome hardware limitations hindering real-time implementation of large transformer models at the edge, (4) seamlessly integrating a heterogeneous system of mobile processors, edge AI accelerators, and neuromorphic hardware for a comprehensive end-to-end solution. Throughout the project, rigorous investigation delves into critical trade-offs between bandwidth, accuracy, performance, and energy consumption.This project is jointly funded by the Software and Hardware Foundation (SHF) core research program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.