CAREER: Frontiers of Distributed Machine Learning with Communication, Computation and Data Constraints

机器学习（ML）在过去十年中取得的巨大成功可以归因于两个关键因素的融合：大计算和大数据。例如，尽管神经网络模型在几十年前就被提出了，但它们只是在负担得起的云计算和大量训练数据集的出现之后才成为主流。分布式训练的最先进方法是集中地洗牌数据，然后将它们划分到配备有强大计算单元和高速通信链路的节点上。这种通信、计算和数据密集型框架的不可或缺性使得资源有限的组织无法使用当前的机器学习算法。该项目旨在通过使ML能够无缝扩展到计算，通信和数据约束节点的网络来实现ML的民主化。预期成果包括分布式训练和推理算法，这些算法是系统感知的（对通信和计算限制具有鲁棒性）和数据感知的（可以处理统计偏差和稀缺数据）。研究成果将由本科生，研究生和高中外展课程和大规模学习的专著补充。该研究员还旨在为女性研究人员举办年度合作研讨会，为STEM领域的女性创造合作和指导机会。该项目包括三个研究方向，分别与通信，计算和数据约束有关。第一个推力开发了分布式训练中通信效率的几个方面，以实现训练时间的数量级减少。第二个重点是解决计算异构性，这可能会导致模型训练和推理中的一致性和可扩展性问题。研究者将设计分布式训练、推理和超参数优化算法，这些算法对这种异质性具有鲁棒性。第三个重点将解决源于数据异构性的基本问题，例如自适应节点选择、公平性、个性化和数据稀缺学习。而不是孤立地改进系统和算法，研究人员将采取系统感知和数据感知的方法，从调度，编码理论和多武装匪徒中获得新的见解。她还将与谷歌的联合学习团队合作，验证研究成果并扩大其影响。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The meteoric success of machine learning (ML) during the past decade can be attributed to the confluence of two key factors: big compute and big data. For example, although neural network models were proposed several decades ago, they came into the mainstream only after the advent of affordable cloud computing, and the availability of massive training datasets. The state-of-the-art approach towards distributed training is to centrally shuffle data and then partition them across nodes equipped with powerful computation units and high-speed communication links. The indispensability of such communication-, compute- and data-intensive frameworks precludes resource-limited organizations from using current ML algorithms. This project seeks to democratize ML by enabling it to seamlessly scale to a network of computation-, communication-, and data-constrained nodes. Expected outcomes include distributed training and inference algorithms that are system-aware (robust to communication and computation limitations) and data-aware (can handle statistically skewed and scarce data). The research outcomes will be complemented by undergraduate, graduate, and high-school outreach classes and a monograph on large-scale learning. The investigator also aims to host an annual collaboration workshop for female researchers to create collaboration and mentorship opportunities for women in STEM.This project consists of three research thrusts related to communication, computation, and data constraints, respectively. The first thrust develops several facets of communication efficiency in distributed training to achieve an order-of-magnitude reduction in training time. The second thrust will tackle computational heterogeneity, which can cause consistency and scalability issues in model training and inference. The investigator will design distributed training, inference, and hyper-parameter optimization algorithms robust to such heterogeneity. The third thrust will address fundamental problems that stem from data heterogeneity, such as adaptive node selection, fairness, personalization, and data-scarce learning. Rather than improving the system and the algorithms in isolation, the investigator will take a system-aware and data-aware approach that draws novel insights from scheduling, coding theory, and multi-armed bandits. She will also collaborate with Google's federated learning team to validate the research outcomes and amplify their impact.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.