Collaborative Research: CNS Core: Small: Towards Unsupervised Learning on Resource Constrained Edge Devices with Novel Statistical Contrastive Learning Scheme

深度学习模型已经部署在越来越多的边缘和移动设备中，以支持我们生活中的各种任务，从智能手机和增强现实(AR)/混合现实(XR)眼镜的个人帮助到医疗机器人。然而，现有部署的一个缺点是，神经网络不适合不同的用户和应用领域，当新的看不见的数据流在云中训练并部署到设备中时，神经网络也不会进化。现有的设备上培训计划都需要手动数据标记，一旦在设备上部署，由于对专业知识、数据隐私、通信成本或延迟的强烈要求，这可能会非常昂贵或具有挑战性。因此，对于设备上的学习模型来说，能够在资源受限的环境中以尽可能少的标签在原位学习新的流数据是更实用和有用的。本项目旨在为无监督的设备上深度学习框架奠定技术基础，在设备上的深度学习模型可以在最少的人工干预下连续学习视觉表征。将执行三项任务，以实现高效的计算和内存利用，以及高学习速度和精度，同时克服流数据中的非独立和相同分布(Non-IID)问题。该项目将与行业合作伙伴Misty Robotics和Facebook在包括机器人、增强现实(AR)和混合现实(XR)在内的目标应用程序上通过真实系统和应用程序进行评估。该项目的成功将为机器学习驱动的设备带来更高的精确度，并为每个人带来更好的用户体验。更重要的是，该项目将通过对新的输入数据进行持续的个性化处理，改善数据收集过程中代表性不足的少数群体的推理性能，从而提高人工智能的公平性。它还将使部署在偏远地区的设备具有学习能力，以便它们能够快速适应新的环境，这将极大地造福于各种消费、商业、科学和国家安全应用，如战场侦察和外层空间探索。拟议研究的教育影响包括基于两个私人投资机构的现有资源整合各种教育活动，如DAC系统设计大赛；通过皮特的现在投资暑期学校和ND的印第安纳州K-12学生的CS课程扩展本地K-12学生；强调少数群体参与的本科生研究，以及研究成果的课程整合。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning models have been deployed in an increasing number of edge and mobile devices to power various tasks in our life, from personal assistance in smartphones and augmented reality (AR)/mixed reality (XR) glasses to healthcare robotics. One drawback of existing deployment, however, is that neural networks do not adapt to different users and application domains, nor do they evolve when new unseen data stream in once trained in the cloud and deployed in the devices. Existing on-device training schemes all require manual data labeling, which can be very expensive or challenging once deployed on devices due to strong requirements on expert knowledge, data privacy, communication cost, or latency. Therefore, it is more practical and useful for on-device learning models to be able to learn from new streaming data in-situ with as few labels as possible, in a resource-constrained environment. This project aims to lay the technological foundation for unsupervised on-device learning framework, in which the on-device deep learning models can continuously learn visual representations with minimal human intervention. Three tasks will be carried out to achieve efficient computation and memory utilization, as well as high learning speed and accuracy while overcoming the non-independent and identically distributed (non-IID) issue in streaming data. This project will be evaluated with real systems and applications with industry collaborators Misty Robotics and Facebook on target applications including robotics, augmented reality (AR) and mixed reality (XR).The success of this project will lead to higher accuracy for machine learning-powered devices and a better user experience for everyone. More importantly, this project will enhance the fairness of AI by improving the inference performance for minorities under-represented in the data collection process, through continuous personalization on new incoming data. It will also enable learning capability for devices deployed in remote areas such that they can quickly adapt to new environments, which will drastically benefit various consumer, business, scientific and national security applications such as battlefield scouting and outer space exploration. The education impacts of the proposed research include the integration of various educational activities based on the resources available to the two PIs such as DAC System Design Contest; outreach for local K-12 students through Pitt’s Investing Now summer school and ND’s CS curriculum for K-12 students in Indiana; undergraduate research with emphasis on minority participation, and course integration of the research outcomes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.