CRII: CIF: Unifying Scheduling and Optimization Techniques to Speed-up Distributed Stochastic Gradient Descent

随机梯度下降（SGD）是最先进的监督学习的核心，它正在彻底改变许多不同应用中的推理和决策，如自动驾驶汽车、机器人、个性化搜索和推荐以及医疗诊断。因此，提高随机梯度下降算法的速度是一个及时而重要的研究课题。由于目前使用的神经网络模型和训练数据集规模庞大，因此在多个计算节点上并行SGD变得非常有利。虽然并行化SGD增加了每次迭代处理的数据量，但它使算法暴露于不可预测的节点减速和通信延迟，这些都是由计算基础设施的变化引起的。该项目的目标是设计可证明的快速SGD算法，这些算法可以轻松地用于分布式实现，并且对计算和网络延迟的波动以及不可预测的节点故障具有鲁棒性。该项目可以帮助机器学习变得普遍可用，而无需访问昂贵的高性能计算基础设施。将发布由此产生的自适应分布式SGD算法的开源实现。研究成果还将被纳入卡内基梅隆大学的两个新的机器学习课程，以及K-12教师和学生的课程开发和研究样本研讨会。单节点SGD的速度通常根据训练误差相对于迭代次数的收敛来衡量。在分布式SGD中，每次迭代的运行时间取决于系统级因素，例如工作节点处的计算延迟和梯度聚合机制。因此，有一个关键的需要，以了解相对于挂钟时间，而不是迭代次数的误差收敛。该项目将通过联合优化每次迭代的运行时间和错误与迭代的关系来提高分布式SGD在挂钟时间方面的真正收敛性。它将考虑两个流行的分布式SGD框架，参数服务器模型和通信高效SGD模型。该研究预计将提供新的运行时和错误分析的分布式SGD在这些框架和设计的第一个自适应分布式SGD算法，罢工的最佳错误运行时trade-off.This奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

Stochastic gradient descent (SGD) is at the core of state-of-the-art supervised learning, which is revolutionizing inference and decision-making in many diverse applications such as self-driving cars, robotics, personalized search and recommendations, and medical diagnosis. Thus, improving the speed of stochastic gradient descent is a timely and important research problem. Due to the massive scale of neural network models and training data sets used today, it has become advantageous to parallelize SGD across multiple computing nodes. Although parallelizing SGD boosts the amount of data processed per iteration, it exposes the algorithm to unpredictable node slowdown and communication delays stemming from variability in the computing infrastructure. The goal of this project is to design provably fast SGD algorithms that easily lend themselves to distributed implementations, and are robust to fluctuations in computation and network delays as well as unpredictable node failures. This project can assist in making machine learning universally accessible, without requiring access to expensive high-performance computing infrastructure. An open-source implementation of the resulting adaptive distributed SGD algorithms will be released. The research outcomes will also be incorporated into two new machine learning classes at Carnegie Mellon University, and into curriculum development and research sampler workshops for K-12 teachers and students.The speed of single-node SGD is typically measured in terms of the convergence of training error with respect to the number of iterations. In distributed SGD, the runtime per iteration depends on system-level factors such as the computation delays at worker nodes and the gradient aggregation mechanism. Thus, there is a critical need to understand the error convergence with respect to the wall-clock time rather than the number of iterations. This project will improve the true convergence of distributed SGD with respect to wall-clock time by jointly optimizing the runtime-per-iteration and error-versus-iterations. It will consider two popular distributed SGD frameworks, the parameter server model and the communication-efficient SGD model. The research is expected to provide novel runtime and error analyses of distributed SGD in these frameworks and design the first adaptive distributed SGD algorithms that strike the best error-runtime trade-off.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.