MM optimization: Proximal distance algorithms, path following, and trust regions.
MM optimization: Proximal distance algorithms, path following, and trust regions.
复制标题
MM优化：最近距离算法、路径跟踪和信任域。
DOI：
10.1073/pnas.2303168120
复制
发表时间：
2023-07-04
影响因子：
11.1
通讯作者：
Lange, Kenneth
中科院分区：
文献类型：
--
作者：
Landeros, Alfonso;Xu, Jason;Lange, Kenneth
优化方法是应用数学、统计学和机器学习的基本方法。通过将一个复杂的高维优化问题转化为一系列较简单的优化问题，优化-最小化(MM)原理是设计新算法的一种通用工具。巧妙地应用分析中的不等式，允许实践者设计易于优化的代理函数，并使MM算法具有所需的特征。例如，MM算法可以分离参数，允许并行更新，或者将更新步骤减少为求解线性方程组。距离优化进一步将MM的适用范围扩展到约束优化问题。我们简要回顾了优选化-最小化(MM)原理，并详细阐述了最近距离算法的密切相关概念，这是一种通过二次惩罚来求解约束优化问题的通用方法。我们说明了MM和近邻距离原理如何应用于统计学、金融学和非线性最优化中的各种问题。从我们选择的例子中，我们还提出了一些与MM算法加速有关的想法：a)围绕有效的矩阵分解构造更新，b)近距离迭代中的路径跟踪，c)三次优化及其与信赖域方法的联系。这些想法在几个数值例子上得到了检验，但为了简洁起见，我们省略了与竞争方法的详细比较。当前的文章混合了回顾和当前的贡献，称赞MM原则是设计优化算法和重新解释现有算法的强大框架。
Optimization methods are essential to applied mathematics, statistics, and machine learning. By converting a difficult high-dimensional optimization problem into a sequence of simpler ones, the majorization–minimization (MM) principle is a versatile tool for designing novel algorithms. Clever application of inequalities from analysis allows practitioners to design surrogate functions that are easy to optimize and imbue an MM algorithm with desirable characteristics. For example, an MM algorithm can separate parameters, allowing for parallel updates, or reduce an update step to solving a system of linear equations. Distance majorization further extends the reach of MM to constrained optimization problems. We briefly review the majorization–minimization (MM) principle and elaborate on the closely related notion of proximal distance algorithms, a generic approach for solving constrained optimization problems via quadratic penalties. We illustrate how the MM and proximal distance principles apply to a variety of problems from statistics, finance, and nonlinear optimization. Drawing from our selected examples, we also sketch a few ideas pertinent to the acceleration of MM algorithms: a) structuring updates around efficient matrix decompositions, b) path following in proximal distance iteration, and c) cubic majorization and its connections to trust region methods. These ideas are put to the test on several numerical examples, but for the sake of brevity, we omit detailed comparisons to competing methods. The current article, which is a mix of review and current contributions, celebrates the MM principle as a powerful framework for designing optimization algorithms and reinterpreting existing ones.
登录
查看更多内容
影响因子：
2.7
作者：
Chi, Eric C.;Zhou, Hua;Lange, Kenneth
通讯作者：
Lange, Kenneth
影响因子：
2.7
作者：
Bien, Jacob;Tibshirani, Robert J.
通讯作者：
Tibshirani, Robert J.
影响因子：
1.2
作者：
Bruni R;Cesarone F;Scozzari A;Tardella F
通讯作者：
Tardella F
影响因子：
1.8
作者：
Hardin, Johanna;Garcia, Stephan Ramon;Golan, David
通讯作者：
Golan, David
DOI：
10.1111/j.2517-6161.1977.tb01600.x
发表时间：
1977-01-01
期刊：
JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-METHODOLOGICAL
影响因子：
--
作者：
DEMPSTER, AP;LAIRD, NM;RUBIN, DB
通讯作者：
RUBIN, DB