Collaborative Research: Consensus and Distributed Optimization in Non-Convex Environments with Applications to Networked Machine Learning

分布式优化是通过网络进行机器学习和数据分析的工具，因为它提供了以可扩展的方式训练模型的方法。本计画旨在借由研究两类重要的非凸性问题，增进分散式非凸性最佳化的基础知识。该项目的智力价值包括研究分布式无收缩流形优化和分布式非光滑弱凸优化。该项目的科学贡献将为非凸环境中的共识和协调的理解带来变革性的变化。该项目更广泛的影响包括教育组件，将分布式优化作为下一代工程师的实用工具。这些教育计划包括面向更广泛受众的研讨会演示以及高级课程开发，以介绍最先进的分散优化技术。该项目的目标是解决异构计算环境中实现非凸优化的分布式方法。将研究用于分布式实现黎曼随机梯度下降的无收缩方法，然后说明该技术在降秩和稀疏回归以及稀疏深度神经网络训练中的优势。在这两种情况下，稀疏性将通过将对模型参数的搜索约束到低维黎曼流形来确保。该研究将进一步考虑分布式非光滑，非凸（弱凸）优化，并开发异步次梯度下降方法的通信效率优化。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Distributed optimization is a vehicle for machine learning and data analysis over networks as it provides the means to train models in a scalable fashion. This project aims to advance the fundamental knowledge on distributed non-convex optimization by studying two important classes of non-convex problems. The intellectual merits of the project include investigating distributed retraction-free manifold optimization, and distributed non-smooth weakly-convex optimization. Scientific contributions of this project will bring transformative change to the understanding of consensus and coordination in non-convex environments. The broader impacts of the project include educational components to introduce distributed optimization as a practical tool for the next generation of engineers. These educational plans include seminar presentations for a broader audience as well as advanced course development to introduce state-of-the-art decentralized optimization techniques.The goal of this project is to address distributed approaches for non-convex optimization implemented in heterogeneous computing environments. Retraction-free methods for distributed implementation of Riemannian Stochastic Gradient Descent will be investigated, followed by illustration of the benefits of this technique in reduced rank and sparse regression as well as training of sparse deep neural networks. In both cases, sparsity will be ensured by constraining the search for the model parameters to a lower-dimensional Riemannian manifold. The study will further consider distributed non-smooth, non-convex (weakly-convex) optimization, and develop asynchronous sub-gradient descent methods for communication-efficient optimization. The goal will be to establish both global and local convergence results for the proposed methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.