Collaborative Research: CIF: Small: A Unified Framework of Distributional Optimization via Variational Transport

分布优化是指目标函数中的优化变量是某一空间上的概率度量的一类数学问题。由于其高度技术性，分布式优化在很大程度上仍未得到探索，仅在特定问题实例上取得了进展。该项目提出了一个统一的框架，以探索在广泛的重要应用领域具有挑战性的分布式优化问题。其主要目标是发展一个全面的理论，支持新的和有效的优化算法的原则性设计。要做到这一点，需要在几个数学学科之间建立联系，包括最优化理论、最优运输、函数不等式和统计学。这将促进思想的交流，并从跨学科的角度编写培训材料。由此产生的开放源码包将用于支持我们日常生活所依赖的相关领域的研究工作。分布优化问题是无限维优化问题，其中优化变量是一个概率度量。许多研究问题都属于这类问题，特别是欧氏空间上的任何非凸优化问题都可以归结为凸分布优化问题。传统上，这些问题的具体实例是相互独立地研究的。将这些看似不同的优化问题组合到一个单一的统一框架中，将允许使用更强大的数学技术和工具。这将导致对解决方案结构的更深入了解，以及为人工智能和数据科学的大规模应用量身定做的高效算法。所提出的框架基于最优传输理论，该理论赋予分布空间一个自然的几何形状。该算法利用了目标相对于该几何形状的梯度流。为了实现可伸缩性，优化变量由一组粒子近似，算法现在描述粒子的集体动力学。一种新的变分方法将被用来逼近梯度下降方向。我们将深入研究该算法的理论性质，包括可证明的性能保证、收敛速度和统计性质。将通过将这一统一框架专门用于贝叶斯推理和分布稳健学习等应用程序来进行案例研究。还将通过结合现有的优化技术(如动量和方差减少)来研究该算法的加速，以此作为提高收敛速度的一种方式。最后，该项目将探索如何将该算法从最小化问题调整为最小-最大问题，以处理博弈论应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Distributional optimization refers to a class of mathematical problems where the optimizing variable in the objective function is a probability measure over some space. Because of its highly technical nature, distributional optimization has remained largely unexplored with advances made only on specific problem instances. This project proposes a unified framework to explore challenging distributional optimization problems in a wide range of important application domains. The main objective is to develop a comprehensive theory supporting the principled design of novel and efficient optimization algorithms. To do so, establishing connections between several mathematical disciplines will be required, including optimization theory, optimal transport, functional inequalities, and statistics. This will promote the cross-fertilization of ideas and lead to the creation of training material from an interdisciplinary perspective. The resulting open-source packages will be made available to support research efforts in related fields that our daily lives depend on. Problems in distributional optimization are infinite-dimensional optimization problems where the optimization variable is a probability measure. Many research problems fall into this class of problems; in particular, any non-convex optimization problem over Euclidean space can be cast as a convex distributional optimization problem. Traditionally, specific instances of these problems have been studied independently of each other. Formulating these seemingly different optimization problems into a single unified framework will allow more powerful mathematical techniques and tools to be used. This will lead to deeper insights into the structure of solutions, and to efficient algorithms tailored to large-scale applications in artificial intelligence and data science. The proposed framework is based on optimal transport theory that endows the space of distributions with a natural geometry. The proposed algorithm utilizes the gradient flow of the objective with respect to this geometry. To achieve scalability, the optimization variable is approximated by a collection of particles, with the algorithm now describing the collective dynamics of the particles. A novel variational approach will be used to approximate the gradient descent direction. The theoretical properties of this algorithm will be investigated thoroughly, including provable performance guarantees, convergence rates, and statistical properties. Case studies will be carried out by specializing this unified framework to applications such as Bayesian inference and distributionally robust learning. The acceleration of this algorithm will also be investigated by incorporating existing optimization techniques, such as momentum and variance reduction, as a way to improve convergence rates. Finally, the project will explore how to adapt this algorithm from minimization problems to min-max problems in order to deal with game-theoretic applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.