Collaborative Research: CNS Core: Small: Towards Unsupervised Learning on Resource Constrained Edge Devices with Novel Statistical Contrastive Learning Scheme

合作研究：CNS 核心：小型：利用新颖的统计对比学习方案在资源受限的边缘设备上实现无监督学习

• 批准号: 2122320
• 负责人: Jingtong Hu
• 金额: $ 25万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2122320&HistoricalAwards=false
• 关键词: Collaborative; Research; CNS; Core; Small

Deep learning models have been deployed in an increasing number of edge and mobile devices to power various tasks in our life, from personal assistance in smartphones and augmented reality (AR)/mixed reality (XR) glasses to healthcare robotics. One drawback of existing deployment, however, is that neural networks do not adapt to different users and application domains, nor do they evolve when new unseen data stream in once trained in the cloud and deployed in the devices. Existing on-device training schemes all require manual data labeling, which can be very expensive or challenging once deployed on devices due to strong requirements on expert knowledge, data privacy, communication cost, or latency. Therefore, it is more practical and useful for on-device learning models to be able to learn from new streaming data in-situ with as few labels as possible, in a resource-constrained environment. This project aims to lay the technological foundation for unsupervised on-device learning framework, in which the on-device deep learning models can continuously learn visual representations with minimal human intervention. Three tasks will be carried out to achieve efficient computation and memory utilization, as well as high learning speed and accuracy while overcoming the non-independent and identically distributed (non-IID) issue in streaming data. This project will be evaluated with real systems and applications with industry collaborators Misty Robotics and Facebook on target applications including robotics, augmented reality (AR) and mixed reality (XR).The success of this project will lead to higher accuracy for machine learning-powered devices and a better user experience for everyone. More importantly, this project will enhance the fairness of AI by improving the inference performance for minorities under-represented in the data collection process, through continuous personalization on new incoming data. It will also enable learning capability for devices deployed in remote areas such that they can quickly adapt to new environments, which will drastically benefit various consumer, business, scientific and national security applications such as battlefield scouting and outer space exploration. The education impacts of the proposed research include the integration of various educational activities based on the resources available to the two PIs such as DAC System Design Contest; outreach for local K-12 students through Pitt’s Investing Now summer school and ND’s CS curriculum for K-12 students in Indiana; undergraduate research with emphasis on minority participation, and course integration of the research outcomes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习模型已被部署在越来越多的边缘和移动的设备中，以支持我们生活中的各种任务，从智能手机和增强现实（AR）/混合现实（XR）眼镜中的个人辅助到医疗机器人。然而，现有部署的一个缺点是，神经网络不能适应不同的用户和应用领域，当新的不可见数据流在云中训练并部署在设备中时，它们也不会进化。 现有的设备上训练方案都需要手动数据标记，由于对专业知识、数据隐私、通信成本或延迟的强烈要求，一旦部署在设备上，这可能非常昂贵或具有挑战性。 因此，在资源受限的环境中，设备上的学习模型能够以尽可能少的标签从新的流数据中原位学习是更实用和有用的。 该项目旨在为无监督的设备上学习框架奠定技术基础，在该框架中，设备上深度学习模型可以在最少的人为干预下持续学习视觉表示。将执行三个任务，以实现高效的计算和内存利用率，以及高学习速度和准确性，同时克服流数据中的非独立同分布（非IID）问题。该项目将与行业合作伙伴Misty Robotics和Facebook一起使用真实的系统和应用程序进行评估，目标应用包括机器人，增强现实（AR）和混合现实（XR）。该项目的成功将导致机器学习驱动的设备具有更高的准确性，并为每个人提供更好的用户体验。更重要的是，该项目将通过对新输入数据的持续个性化来提高数据收集过程中代表性不足的少数群体的推理性能，从而提高人工智能的公平性。它还将为部署在偏远地区的设备提供学习能力，使其能够快速适应新环境，这将极大地有利于各种消费者，商业，科学和国家安全应用，如战场侦察和外太空探索。拟议研究的教育影响包括根据两个PI可用的资源整合各种教育活动，例如DAC系统设计比赛;通过Pitt的Investing Now暑期学校和ND为印第安纳州的K-12学生提供的CS课程，为当地K-12学生提供外展服务;本科生研究，重点是少数民族的参与，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响进行评估来支持审查标准。

项目成果

Enabling Weakly Supervised Temporal Action Localization From On-Device Learning of the Video Stream

• DOI: 10.1109/tcad.2022.3197536
• 发表时间: 2022-08
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Yue Tang;Yawen Wu;Peipei Zhou;Jingtong Hu

Enabling On-Device Self-Supervised Contrastive Learning with Selective Data Contrast

通过选择性数据对比实现设备上自我监督对比学习

• DOI: 10.1109/dac18074.2021.9586228
• 发表时间: 2021
• 期刊: 2021 58th ACM/IEEE Design Automation Conference (DAC
• 作者: Wu, Yawen;Wang, Zhepeng;Zeng, Dewen;Shi, Yiyu;Hu, Jingtong
• 通讯作者: Hu, Jingtong

Synthetic Data Can Also Teach: Synthesizing Effective Data for Unsupervised Visual Representation Learning

• DOI: 10.1609/aaai.v37i3.25388
• 发表时间: 2022-02
• 作者: Yawen Wu;Zhepeng Wang;Dewen Zeng;Yiyu Shi;Jingtong Hu

Self-Supervised On-Device Federated Learning From Unlabeled Streams

• DOI: 10.1109/tcad.2023.3274956
• 发表时间: 2022-12
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Jiahe Shi;Yawen Wu;Dewen Zeng;Jun Tao;Jingtong Hu;Yiyu Shi

Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learning (Invited Paper)

通过设备端学习进行皮肤病诊断的联邦对比学习（特邀论文）

• DOI: 10.1109/iccad51958.2021.9643454
• 发表时间: 2021
• 期刊: 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD
• 作者: Wu, Yawen;Zeng, Dewen;Wang, Zhepeng;Sheng, Yi;Yang, Lei;James, Alaina J.;Shi, Yiyu;Hu, Jingtong
• 通讯作者: Hu, Jingtong