CAREER: Strengthening the Theoretical Foundations of Federated Learning: Utilizing Underlying Data Statistics in Mitigating Heterogeneity and Client Faults

将受益于改进的机器学习的现实世界应用程序涵盖了广泛的行业和领域，如医疗保健、自动驾驶汽车、自然语言处理以及制造业和工业。近年来，由于对实时数据处理、低延迟和隐私问题的需求不断增加，分布式机器学习获得了显著的发展势头。边缘设备的快速发展扩大了分布式机器学习的适用性，但也带来了巨大的挑战，需要重新审视联邦学习的基本原理和算法设计。该项目的研究目标是巩固分布式机器学习的理论基础和丰富算法工具箱，重点是增强其对广泛的数据异构性、系统缺陷(或故障)和外部攻击的弹性。该项目的教育目标是在下一代机器学习从业者和研究人员中宣传原则性数学思维对于解决机器学习中现实世界问题的重要性，重点是开发针对妇女和代表性不足的少数群体的计划。联合学习是一种快速发展的分布式机器学习方法，它简化了全局模型训练，而不需要共享原始的局部数据。现有的大多数联合学习的理论分析都是从优化的角度出发的，但大多忽略了数据集的潜在统计结构。这经常导致悲观的理论预测与经验成功之间的错位。此外，最近的工作表明，联合学习分析中经常采用的有界梯度相异条件对于实际应用来说可能过于悲观。受我们初步成功的激励和大量先前工作的支持，这项提议旨在通过利用联合数据集的底层结构和设计新的算法来加强联合学习的理论基础，并增强其对广泛的数据异构性和系统故障的弹性。为了实现这一目标，我们将使用和创新统计学习、分布式计算、高维概率和优化等工具。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Real-world applications that would benefit from improved machine learning encompass a wide range of industries and domains such as healthcare, autonomous vehicles, natural language processing, and manufacturing and industry. Distributed machine learning has gained significant momentum in recent years due to the increasing need for real-time data processing, low latency, and privacy concerns. The rapid development of edge devices broadens the applicability of distributed machine learning yet brings nontrivial challenges that call for revisiting the fundamental principles and algorithm designs for federated learning. The research goal of this project is to consolidate the theoretical foundations and to enrich the algorithmic toolbox of distributed machine learning with a focus on enhancing its resilience against a wide range of data heterogeneity, system imperfection (or faults), and external attacks. The educational objective of this project is to promote the importance of principled mathematical thinking for solving real-world problems in machine learning among the next generation of machine learning practitioners and researchers, with a focus on developing programs that target women and underrepresented minority groups. Federated learning is a rapidly evolving distributed machine learning approach that facilitates global model training without the necessity of sharing raw local data. Most existing theoretical analysis of federated learning is derived from an optimization perspective but the underlying statistical structure of the dataset is mostly overlooked. This often leads to misalignment between the pessimistic theoretical predictions and empirical success. In addition, recent work suggests that the bounded gradient dissimilarity conditions, which are frequently adopted in federated learning analysis, may be too pessimistic for practical applications. Motivated by our preliminary successes and backed by extensive prior work, this proposal aims to strengthen the theoretical foundations of federated learning and to enhance its resilience against a wide range of data heterogeneity and system failures, by leveraging the underlying structures of the federated datasets and by designing new algorithms. Towards this goal we will employ and innovate tools from statistical learning, distributed computing, high-dimensional probabilities, and optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.