CRII: CIF: Unifying Scheduling and Optimization Techniques to Speed-up Distributed Stochastic Gradient Descent

随机梯度下降（SGD）是最先进的监督学习的核心，它在许多不同的应用中彻底改变了推理和决策，如自动驾驶汽车、机器人、个性化搜索和推荐以及医疗诊断。因此，提高随机梯度下降的速度是一个及时而重要的研究问题。由于目前使用的神经网络模型和训练数据集的规模庞大，跨多个计算节点并行化SGD变得非常有利。尽管并行化SGD提高了每次迭代处理的数据量，但它使算法暴露于计算基础设施中的可变性所导致的不可预测的节点减速和通信延迟。该项目的目标是设计可证明的快速SGD算法，使其易于分布式实现，并且对计算波动和网络延迟以及不可预测的节点故障具有鲁棒性。这个项目可以帮助机器学习普及，而不需要使用昂贵的高性能计算基础设施。由此产生的自适应分布式SGD算法的开源实现将被发布。研究成果还将被纳入卡内基梅隆大学的两个新的机器学习课程，并纳入K-12教师和学生的课程开发和研究样本研讨会。单节点SGD的速度通常是根据训练误差相对于迭代次数的收敛来衡量的。在分布式SGD中，每次迭代的运行时取决于系统级因素，例如工作节点上的计算延迟和梯度聚合机制。因此，非常需要了解与时钟时间相关的误差收敛，而不是迭代次数。这个项目将通过联合优化每次迭代的运行时间和每次迭代的错误时间来改进分布式SGD在时钟时间方面的真正收敛性。它将考虑两种流行的分布式SGD框架，参数服务器模型和通信高效的SGD模型。该研究有望在这些框架中为分布式SGD提供新颖的运行时和错误分析，并设计出第一个实现最佳错误-运行时权衡的自适应分布式SGD算法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Stochastic gradient descent (SGD) is at the core of state-of-the-art supervised learning, which is revolutionizing inference and decision-making in many diverse applications such as self-driving cars, robotics, personalized search and recommendations, and medical diagnosis. Thus, improving the speed of stochastic gradient descent is a timely and important research problem. Due to the massive scale of neural network models and training data sets used today, it has become advantageous to parallelize SGD across multiple computing nodes. Although parallelizing SGD boosts the amount of data processed per iteration, it exposes the algorithm to unpredictable node slowdown and communication delays stemming from variability in the computing infrastructure. The goal of this project is to design provably fast SGD algorithms that easily lend themselves to distributed implementations, and are robust to fluctuations in computation and network delays as well as unpredictable node failures. This project can assist in making machine learning universally accessible, without requiring access to expensive high-performance computing infrastructure. An open-source implementation of the resulting adaptive distributed SGD algorithms will be released. The research outcomes will also be incorporated into two new machine learning classes at Carnegie Mellon University, and into curriculum development and research sampler workshops for K-12 teachers and students.The speed of single-node SGD is typically measured in terms of the convergence of training error with respect to the number of iterations. In distributed SGD, the runtime per iteration depends on system-level factors such as the computation delays at worker nodes and the gradient aggregation mechanism. Thus, there is a critical need to understand the error convergence with respect to the wall-clock time rather than the number of iterations. This project will improve the true convergence of distributed SGD with respect to wall-clock time by jointly optimizing the runtime-per-iteration and error-versus-iterations. It will consider two popular distributed SGD frameworks, the parameter server model and the communication-efficient SGD model. The research is expected to provide novel runtime and error analyses of distributed SGD in these frameworks and design the first adaptive distributed SGD algorithms that strike the best error-runtime trade-off.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.