Towards Constituting Mathematical Structures for Learning to Optimize
Towards Constituting Mathematical Structures for Learning to Optimize
复制标题
DOI：
10.48550/arxiv.2305.18577
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Jialin Liu;Xiaohan Chen;Zhangyang Wang;W. Yin;HanQin Cai
中科院分区：
文献类型：
--
作者：
Jialin Liu;Xiaohan Chen;Zhangyang Wang;W. Yin;HanQin Cai
学习优化（L2O）是一种利用机器学习从数据中自动学习优化算法的技术，近年来受到越来越多的关注。一种通用的L2O方法参数化迭代更新规则，并以黑盒网络的形式学习更新方向。虽然一般方法是广泛适用的，但学习到的模型可能会过拟合，并且可能不能很好地泛化到分布外测试集。本文导出了成功更新规则通常满足的基本数学条件。因此，我们提出了一个具有数学启发结构的新型L2O模型，该模型广泛适用于分布外问题。数值模拟验证了我们的理论发现，并证明了所提出的L2O模型具有优越的经验性能。
Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. In this paper, we derive the basic mathematical conditions that successful update rules commonly satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems. Numerical simulations validate our theoretical findings and demonstrate the superior empirical performance of the proposed L2O model.