III:Medium:Computation and Communication Efficient Distributed Learning

随着大规模机器学习任务的爆炸性增长和计算资源的日益可获得性，分布式学习已经成为从大数据中提取信息和知识的基石。分布式学习中的节点需要进行信息交流。因此，分布式学习面临着类似于传统机器学习的计算效率方面的挑战。但它也面临着沟通效率的额外挑战。这些效率问题极大地阻碍了分布式学习在大规模机器学习任务和资源受限的边缘计算等复杂计算环境中的应用。在这个项目中，我们迎接新的挑战和机遇，全面研究分布式学习中的计算和通信效率。该项目的新颖性为在通信协议和网络带宽有限的大规模计算集群中开发和部署高效和可扩展的分布式学习算法提供了新的视角。在大数据无处不在的今天，该项目的影响是造福于来自计算机科学、社会科学等领域的许多现实世界的应用。该项目旨在从效率的角度解决现有分布式学习算法的主要缺陷，并极大地提高大规模分布式学习的效率和可扩展性。为了实现这一目标，我们系统地研究了集中式和分散式两种分布式学习范式，以及计算效率和通信效率这两大效率障碍。为了解决这些范式和障碍，该项目有三个专门设计的研究方向。每个方向都将极大地扩展科学，不仅提供严格的理论保证，而且还在实践系统中进行全面的实证研究。核心知识是对科学的全面调查和新方法的设计，以加深我们对分布式学习系统和算法的效率、可扩展性和实用用途的理解。该项目的成果将是：(1)新的高效和可扩展的分布式学习算法，具有最先进的计算和通信效率以及预测精度；(2)理论分析，如收敛速度和通信复杂性；以及(3)所有关键算法、系统和框架的开源实现。这项拟议的研究将让研究生和本科生参与他们的论文或荣誉项目。这个项目的发现和研究成果将紧密地整合到几门现有的和新的课程中。将创建教学内容，以使我们的结果能够快速传播给广泛的受众，并将建立工具，以帮助机器学习知识的认识和采用。该项目的成果将通过多种方式及时传播，如分布式学习资源库、期刊和会议出版物、在知名会议上联合举办的特殊用途教程和研讨会，以及实习生等行业参与。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

With the explosion of large-scale machine learning tasks and the increasing availability of computational resources, distributed learning has become the cornerstone for extracting information and knowledge from big data. Nodes in distributed learning need to communicate information. Thus, distributed learning faces challenges around computational efficiency similar to conventional machine learning. But it also faces the additional challenge of communication efficiency. These efficiency problems have greatly hindered the applications of distributed learning in large-scale machine learning tasks and complex computing environments, such as resource-limited edge computing. In this project, we embrace new challenges and opportunities to comprehensively study the computation and communication efficiency in distributed learning. The project’s novelties are providing new perspectives for developing and deploying efficient and scalable distributed learning algorithms in large-scale computing clusters with limited communication protocols and network bandwidth. Nowadays, as big data is ubiquitous, the project's impacts are to benefit many real-world applications from various disciplines such as computer science, social sciences and others areas.This project aims to tackle the major drawbacks in existing distributed learning algorithms from the efficiency perspective and greatly promote the efficiency and scalability of large-scale distributed learning. To achieve this goal, we systematically investigate two distributed learning paradigms, centralized and decentralized learning, as well as two major efficiency obstacles, computation and communication efficiency. To address these paradigms and obstacles, the project has three dedicated designed research directions. Each direction will dramatically extend the science through not only providing rigorous theoretical guarantees, but also comprehensive empirical studies in practical systems. The core intellectual is a comprehensive investigation on science and the design of novel methodologies to deepen our understanding on the efficiency, scalability, and practical usages of distributed learning systems and algorithms. The outcomes of this project will be: (1) New efficient and scalable distributed learning algorithms with state-of-the-art computation and communication efficiency, as well as predictive accuracy; (2) Theoretical analysis such as convergence rate and communication complexity; and (3) Open-source implementations of all key algorithms, systems, and frameworks. The proposed research will involve graduate and undergraduate students in pursuing their thesis or honor's projects. Discoveries and research findings of this project will be tightly integrated into several current and new courses. Instructional content will be created to enable fast distribution of our results to a wide audience, and tools will be built to help machine learning knowledge awareness and adoption. The findings of this project will be timely disseminated via multiple means such as a distributed learning repository, journal and conference publications, special purpose tutorials and workshops co-held at prominent conferences, and industrial participation such as internships.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.