III: Small: Multiple Device Collaborative Learning in Real Heterogeneous and Dynamic Environments

在协作学习中，不同的设备（如智能手机）或银行或医院等组织一起学习，使用他们自己的（有时是私人的）数据来构建共享模型。该项目解决了将这种学习扩展到大型、不断变化和多样化的数据集的挑战。它建议从传统的计算系统转向更灵活的系统，这些系统可以处理不断变化的数据类型、分散的计算以及学习设备之间的不同策略。目前，问题来自于真实世界数据的多样性和数量，如用户照片、监控视频或医疗信息。目前的协作学习方案往往试图平均所有设备的学习更新，而忽略了每个设备的个体特征及其独特的计算能力。此外，这些系统还面临着“搭便车”的问题，即一些参与者在不提供任何数据的情况下，从改进的学习模式中受益。这一困难需要创造合适的激励机制来鼓励平等参与。该项目的目标是通过解决这些问题来改善协作学习，从而产生更有效和公平的学习系统，以满足个人设备的独特特性。这一进步不仅改善了机器学习，还鼓励了数据共享、参与和包容性，带来了更广泛的社会效益。当前的协作学习框架通常通过平均参与代理的模型更新来达成共识，这种方法可能忽略了所涉及的代理的独特属性和不同的硬件能力。这样的疏忽可能导致模型架构和特定设备的硬件功能之间的不匹配，特别是内存或计算能力有限的边缘设备，从而阻碍有效的模型训练或可用资源的利用不足。此外，现有的协作学习框架往往忽略了算法可信度和机制设计等关键因素。这些挑战突出了对协作学习方法的重新构想的迫切需要。该项目侧重于开发一个专门为动态和多样化环境设计的协作学习框架，特别强调标准硬件计算平台，例如那些包含边缘设备的平台。该项目的目标有三个方面：首先，通过设计独特的模型架构和高效的算法，以采用结构化变分推理方法的理论探索为基础，寻求创新模型并行协作学习。其次，该项目旨在促进创建实用、高效的培训和通信学习算法，以供设备上使用。这一目标将通过引入新的算法组件和一个真正的设备上测试平台来实现。最后，项目拟在去中心化环境下评估系统的可信度和机制设计。它将设计激励措施，促进数据共享和算法采用，从而在社区和个人层面上实现利益最大化。该项目的目标应用包括灾害预测、人工智能辅助临床诊断和治疗，以及分散的战略决策。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In collaborative learning, different devices such as smartphones or organizations like banks or hospitals learn together, using their own (sometimes private) data to build a shared model. This project tackles the challenge of scaling up this kind of learning for large, constantly changing, and diverse datasets. It proposes a move away from traditional computing systems, towards more flexible systems that can handle changing data types, decentralized computing, and different strategies between learning devices. Currently, problems arise from the diverse nature and volume of real-world data, like user photos, surveillance videos, or medical information. Present collaborative learning schemes often try to average out the learning updates from all devices, ignoring the individual characteristics of each device and their unique computing abilities. Moreover, these systems struggle with "free riding", where some participants benefit from the improved learning model without contributing any data. This difficulty calls for creating fitting incentives to encourage equal participation. The goal of this project is to improve collaborative learning by solving these issues, resulting in more efficient and fair learning systems that cater to individual devices' unique characteristics. This advancement goes beyond improving machine learning as it encourages data sharing, participation, and inclusivity, bringing about broader societal benefits. Current collaborative learning frameworks often achieve consensus by averaging model updates from participating agents, an approach that may disregard the unique attributes and diverse hardware capabilities of the agents involved. Such an oversight could lead to a mismatch between the model architecture and the hardware capabilities of specific devices, particularly edge devices with limited memory or computational power, thereby impeding efficient model training or underutilizing available resources. Additionally, the extant collaborative learning frameworks tend to overlook crucial factors such as algorithm trustworthiness and mechanism design. These challenges highlight the urgent need for a reimagined approach to collaborative learning. This project focuses on the development of a collaborative learning framework specifically designed for dynamic and diverse environments, with particular emphasis on standard hardware computing platforms, such as those comprising edge devices. The project's objectives are threefold: First, it seeks to innovate model-parallel collaborative learning by designing unique model architectures and efficient algorithms, underpinned by theoretical explorations employing a structured variational inference approach. Second, the project aims to facilitate the creation of practical, efficient training and communication learning algorithms for on-device usage. This aim will be achieved by introducing new algorithmic components and an authentic on-device testing platform. Lastly, the project intends to evaluate the system's trustworthiness and mechanism design in a decentralized setting. It will design incentives that promote data sharing and algorithm adoption, thereby maximizing benefits at both the community and individual levels. The project's targeted applications encompass disaster forecasting, AI-assisted clinical diagnosis, and treatment, and decentralized strategic decision-making.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.