Collaborative Research: Consensus and Distributed Optimization in Non-Convex Environments with Applications to Networked Machine Learning

分布式优化是通过网络进行机器学习和数据分析的工具，因为它提供了以可扩展的方式训练模型的方法。本项目旨在通过研究两类重要的非凸问题来提高分布式非凸优化的基础知识。该项目的智力优势包括研究分布式无伸缩流形优化和分布式非光滑弱凸优化。该项目的科学贡献将为非凸环境中的共识和协调的理解带来革命性的变化。该项目更广泛的影响包括教育组件，将分布式优化作为下一代工程师的实用工具。这些教育计划包括面向更广泛受众的研讨会演讲，以及介绍最先进的分散优化技术的高级课程开发。这个项目的目标是解决在异构计算环境中实现的非凸优化的分布式方法。将研究黎曼随机梯度下降分布式实现的无回缩方法，然后说明该技术在降秩和稀疏回归以及稀疏深度神经网络训练中的好处。在这两种情况下，通过将模型参数的搜索限制到低维黎曼流形来确保稀疏性。研究将进一步考虑分布式非光滑、非凸（弱凸）优化，并开发异步亚梯度下降方法进行通信高效优化。目标是为所提出的方法建立全局和局部收敛结果。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Distributed optimization is a vehicle for machine learning and data analysis over networks as it provides the means to train models in a scalable fashion. This project aims to advance the fundamental knowledge on distributed non-convex optimization by studying two important classes of non-convex problems. The intellectual merits of the project include investigating distributed retraction-free manifold optimization, and distributed non-smooth weakly-convex optimization. Scientific contributions of this project will bring transformative change to the understanding of consensus and coordination in non-convex environments. The broader impacts of the project include educational components to introduce distributed optimization as a practical tool for the next generation of engineers. These educational plans include seminar presentations for a broader audience as well as advanced course development to introduce state-of-the-art decentralized optimization techniques.The goal of this project is to address distributed approaches for non-convex optimization implemented in heterogeneous computing environments. Retraction-free methods for distributed implementation of Riemannian Stochastic Gradient Descent will be investigated, followed by illustration of the benefits of this technique in reduced rank and sparse regression as well as training of sparse deep neural networks. In both cases, sparsity will be ensured by constraining the search for the model parameters to a lower-dimensional Riemannian manifold. The study will further consider distributed non-smooth, non-convex (weakly-convex) optimization, and develop asynchronous sub-gradient descent methods for communication-efficient optimization. The goal will be to establish both global and local convergence results for the proposed methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.