III:Medium:Computation and Communication Efficient Distributed Learning

随着大规模机器学习任务的爆炸式增长和计算资源的可用性不断提高，分布式学习已成为从大数据中提取信息和知识的基石。分布式学习中的节点需要传递信息。因此，分布式学习面临着与传统机器学习类似的计算效率挑战。但它也面临着通信效率的额外挑战。这些效率问题极大地阻碍了分布式学习在大规模机器学习任务和复杂计算环境中的应用，如资源有限的边缘计算。在这个项目中，我们拥抱新的挑战和机遇，全面研究分布式学习中的计算和通信效率。该项目的新颖性为在具有有限通信协议和网络带宽的大规模计算集群中开发和部署高效和可扩展的分布式学习算法提供了新的视角。在大数据无处不在的今天，该项目的影响将惠及计算机科学、社会科学等多个学科领域的实际应用，旨在从效率的角度解决现有分布式学习算法的主要缺陷，极大地提高大规模分布式学习的效率和可扩展性。为了实现这一目标，我们系统地研究了两种分布式学习范式，集中式和分散式学习，以及两个主要的效率障碍，计算和通信效率。为了解决这些范式和障碍，该项目有三个专门设计的研究方向。每个方向将通过不仅提供严格的理论保证，而且在实际系统中进行全面的实证研究来极大地扩展科学。核心知识是对科学的全面调查和新颖方法的设计，以加深我们对分布式学习系统和算法的效率，可扩展性和实际用途的理解。该项目的成果将是：（1）新的高效和可扩展的分布式学习算法，具有最先进的计算和通信效率，以及预测准确性;（2）理论分析，如收敛速度和通信复杂性;（3）所有关键算法，系统和框架的开源实现。拟议的研究将涉及研究生和本科生在追求他们的论文或荣誉的项目。该项目的发现和研究成果将紧密结合到几个现有的和新的课程。将创建教学内容，以便将我们的结果快速分发给广大受众，并将构建工具来帮助机器学习知识的认知和采用。该项目的研究结果将通过多种方式及时传播，例如分布式学习库、期刊和会议出版物、在著名会议上共同举办的特殊目的教程和研讨会以及实习等行业参与。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

With the explosion of large-scale machine learning tasks and the increasing availability of computational resources, distributed learning has become the cornerstone for extracting information and knowledge from big data. Nodes in distributed learning need to communicate information. Thus, distributed learning faces challenges around computational efficiency similar to conventional machine learning. But it also faces the additional challenge of communication efficiency. These efficiency problems have greatly hindered the applications of distributed learning in large-scale machine learning tasks and complex computing environments, such as resource-limited edge computing. In this project, we embrace new challenges and opportunities to comprehensively study the computation and communication efficiency in distributed learning. The project’s novelties are providing new perspectives for developing and deploying efficient and scalable distributed learning algorithms in large-scale computing clusters with limited communication protocols and network bandwidth. Nowadays, as big data is ubiquitous, the project's impacts are to benefit many real-world applications from various disciplines such as computer science, social sciences and others areas.This project aims to tackle the major drawbacks in existing distributed learning algorithms from the efficiency perspective and greatly promote the efficiency and scalability of large-scale distributed learning. To achieve this goal, we systematically investigate two distributed learning paradigms, centralized and decentralized learning, as well as two major efficiency obstacles, computation and communication efficiency. To address these paradigms and obstacles, the project has three dedicated designed research directions. Each direction will dramatically extend the science through not only providing rigorous theoretical guarantees, but also comprehensive empirical studies in practical systems. The core intellectual is a comprehensive investigation on science and the design of novel methodologies to deepen our understanding on the efficiency, scalability, and practical usages of distributed learning systems and algorithms. The outcomes of this project will be: (1) New efficient and scalable distributed learning algorithms with state-of-the-art computation and communication efficiency, as well as predictive accuracy; (2) Theoretical analysis such as convergence rate and communication complexity; and (3) Open-source implementations of all key algorithms, systems, and frameworks. The proposed research will involve graduate and undergraduate students in pursuing their thesis or honor's projects. Discoveries and research findings of this project will be tightly integrated into several current and new courses. Instructional content will be created to enable fast distribution of our results to a wide audience, and tools will be built to help machine learning knowledge awareness and adoption. The findings of this project will be timely disseminated via multiple means such as a distributed learning repository, journal and conference publications, special purpose tutorials and workshops co-held at prominent conferences, and industrial participation such as internships.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.