Collaborative Research: CNS Core: Small: Towards Unsupervised Learning on Resource Constrained Edge Devices with Novel Statistical Contrastive Learning Scheme

合作研究：CNS 核心：小型：利用新颖的统计对比学习方案在资源受限的边缘设备上实现无监督学习

• 批准号: 2122320
• 负责人: Jingtong Hu
• 金额: $ 25万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2122320&HistoricalAwards=false
• 关键词: Collaborative; Research; CNS; Core; Small

Deep learning models have been deployed in an increasing number of edge and mobile devices to power various tasks in our life, from personal assistance in smartphones and augmented reality (AR)/mixed reality (XR) glasses to healthcare robotics. One drawback of existing deployment, however, is that neural networks do not adapt to different users and application domains, nor do they evolve when new unseen data stream in once trained in the cloud and deployed in the devices. Existing on-device training schemes all require manual data labeling, which can be very expensive or challenging once deployed on devices due to strong requirements on expert knowledge, data privacy, communication cost, or latency. Therefore, it is more practical and useful for on-device learning models to be able to learn from new streaming data in-situ with as few labels as possible, in a resource-constrained environment. This project aims to lay the technological foundation for unsupervised on-device learning framework, in which the on-device deep learning models can continuously learn visual representations with minimal human intervention. Three tasks will be carried out to achieve efficient computation and memory utilization, as well as high learning speed and accuracy while overcoming the non-independent and identically distributed (non-IID) issue in streaming data. This project will be evaluated with real systems and applications with industry collaborators Misty Robotics and Facebook on target applications including robotics, augmented reality (AR) and mixed reality (XR).The success of this project will lead to higher accuracy for machine learning-powered devices and a better user experience for everyone. More importantly, this project will enhance the fairness of AI by improving the inference performance for minorities under-represented in the data collection process, through continuous personalization on new incoming data. It will also enable learning capability for devices deployed in remote areas such that they can quickly adapt to new environments, which will drastically benefit various consumer, business, scientific and national security applications such as battlefield scouting and outer space exploration. The education impacts of the proposed research include the integration of various educational activities based on the resources available to the two PIs such as DAC System Design Contest; outreach for local K-12 students through Pitt’s Investing Now summer school and ND’s CS curriculum for K-12 students in Indiana; undergraduate research with emphasis on minority participation, and course integration of the research outcomes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习模型已被部署在越来越多的优势和移动设备中，以为我们生活中的各种任务提供动力，从智能手机和增强现实（AR）/混合现实（XR）眼镜到医疗保健机器人技术。但是，现有部署的一个缺点是，神经网络不能适应不同的用户和应用程序域，当新的看不见的数据流曾经在云中训练并部署在设备中时，它们也不会发展。现有的设备培训方案都需要手动数据标签，由于对专家知识，数据隐私，沟通成本或延迟的强烈要求，一旦在设备上部署在设备上可能非常昂贵或挑战。因此，在设备学习模型中能够在资源约束环境中以尽可能少的标签从新的流数据中学习，这是更实用和有用的。该项目旨在为无监督的设备学习框架奠定技术基础，在该框架中，在此框架中，深度学习模型可以通过最少的人类干预不断地学习视觉表示。将执行三个任务，以实现有效的计算和内存利用，以及高度学习速度和准确性，同时克服流数据中的非独立和相同分布的（非IID）问题。该项目将通过实际系统和行业合作者Misty Robotics和Facebook在包括机器人技术，增强现实（AR）和混合现实（XR）等目标应用程序上进行评估。该项目的成功将导致机器学习驱动的设备以及每个人更好的用户体验的精度。更重要的是，该项目将通过在新传入数据上连续个性化来提高数据收集过程中少数群体的推理表现，从而提高AI的公平性。它还将为部署在偏远地区的设备提供学习能力，以便它们可以迅速适应新环境，这将极大地使各种消费者，商业，科学和国家安全应用（如《战地风云》侦察和外在太空探索）受益。拟议研究的教育影响包括基于两个PI的资源（例如DAC系统设计竞赛）的各种教育活动的整合；通过Pitt的Investing Now Summer School和ND的CS课程为印第安纳州的K-12学生提供宣传；本科研究重点是少数群体的参与以及研究成果的课程整合。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响评估标准，认为通过评估被认为是珍贵的支持。

项目成果

Enabling Weakly Supervised Temporal Action Localization From On-Device Learning of the Video Stream

• DOI: 10.1109/tcad.2022.3197536
• 发表时间: 2022-08
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Yue Tang;Yawen Wu;Peipei Zhou;Jingtong Hu

Enabling On-Device Self-Supervised Contrastive Learning with Selective Data Contrast

通过选择性数据对比实现设备上自我监督对比学习

• DOI: 10.1109/dac18074.2021.9586228
• 发表时间: 2021
• 期刊: 2021 58th ACM/IEEE Design Automation Conference (DAC
• 作者: Wu, Yawen;Wang, Zhepeng;Zeng, Dewen;Shi, Yiyu;Hu, Jingtong
• 通讯作者: Hu, Jingtong

Self-Supervised On-Device Federated Learning From Unlabeled Streams

• DOI: 10.1109/tcad.2023.3274956
• 发表时间: 2022-12
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Jiahe Shi;Yawen Wu;Dewen Zeng;Jun Tao;Jingtong Hu;Yiyu Shi

Synthetic Data Can Also Teach: Synthesizing Effective Data for Unsupervised Visual Representation Learning

• DOI: 10.1609/aaai.v37i3.25388
• 发表时间: 2022-02
• 作者: Yawen Wu;Zhepeng Wang;Dewen Zeng;Yiyu Shi;Jingtong Hu

EF-Train: Enable Efficient On-device CNN Training on FPGA through Data Reshaping for Online Adaptation or Personalization

• DOI: 10.1145/3505633
• 发表时间: 2022-02
• 期刊: ACM Transactions on Design Automation of Electronic Systems (TODAES)
• 作者: Yue Tang;Xinyi Zhang;Peipei Zhou;Jingtong Hu