Accelerated distributed stochastic optimization methods and applications in machine learning

机器学习，特别是深度学习，已经在广泛的应用中变得越来越有影响力，包括人脸识别、数字图像分类、自然语言处理、自动驾驶汽车和科学计算。深度学习的成功很大程度上取决于大量数据的可用性。一方面，这个“大”数据能够成功地学习数据的潜在分布，因此学习的模型可以在遵循类似分布的新数据点上产生高预测精度。另一方面，庞大的数据量给设计高效的数值方法带来了巨大的挑战。该项目侧重于从计算和数学的角度解决分布式“大”数据所带来的挑战，这些数据可能包含私人信息。本项目的研究成果将纳入研究生水平的专题课程，本科生和研究生将在本领域进行培训并参与本研究，每周将组织一次研讨会，交流与本项目相关的想法。将开发新的计算方法，用于在计算节点集群上训练机器学习模型，以及解决具有处理耦合约束能力的分散多智能体优化问题。主要目标是设计快速收敛和通信高效的优化方法，并为解决大规模分布式机器学习问题提供理论保证。加速压缩近端随机梯度方法将被设计用于解决分布式复合光滑随机问题，加速压缩随机次梯度方法将被设计用于解决分布式非凸非光滑问题，最优分散随机梯度方法将被设计用于解决具有非线性耦合约束的多智能体优化问题。为了提高并行化速度，所提出的方法将采用异步实现。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning, and in particular, deep learning, has become increasingly impactful in a wide range of applications, including face recognition, digital image classification, natural language processing, self-driving vehicles, and scientific computing. The success of deep learning largely depends on the availability of a huge amount of data. This "big" data, on one hand, enables successful learning of the underlying distributions of the data, and thus the learned model can yield high prediction accuracy on new data points that follow similar distributions. On the other hand, the huge amount of data raises great challenges when designing efficient numerical approaches. This project focuses on addressing the challenges that are caused by distributed "big" data that can contain private information, from the computational and mathematical perspectives. Research findings from this project will be included in graduate-level topics courses, undergraduate and graduate students will be trained in this field and will participate in this research, and a weekly seminar will be organized to exchange ideas relevant to this project. New computational approaches will be developed for training machine learning models on a cluster of computing nodes as well as solving decentralized multi-agent optimization problems that have the capacity to handle coupling constraints. The main goal is to design fast-convergent and communication-efficient optimization methods with theoretical guarantees for solving large-scale distributed machine learning problems. Accelerated compressed proximal stochastic gradient methods will be designed for distributed composite smooth stochastic problems, accelerated compressed stochastic subgradient methods will be designed for distributed nonconvex nonsmooth problems, optimal decentralized stochastic gradient methods will be designed for solving multi-agent optimization with nonlinear coupling constraints, and asynchronous implementations will be performed in the proposed methods in order to have high parallelization speed up.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.