M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation
M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation
复制标题
DOI：
10.48550/arxiv.2303.00039
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Junjie Yang;Xuxi Chen;Tianlong Chen;Zhangyang Wang;Yitao Liang
中科院分区：
文献类型：
--
作者：
Junjie Yang;Xuxi Chen;Tianlong Chen;Zhangyang Wang;Yitao Liang
学习优化（L2 O）已引起越来越多的关注，因为它往往显着加快复杂任务的优化过程中“过拟合”特定的任务类型，导致性能增强相比，分析优化器。通常，L2 O开发了参数化优化方法（即，"optimizer”）通过解决示例问题进行学习。这种数据驱动的过程产生了L2 O，可以有效地解决类似于训练中看到的问题，即从相同的“任务分布”中提取。然而，当新的测试问题与训练任务分布有很大的偏差时，这种学习优化器经常会遇到困难。本文研究了一个潜在的解决方案，这个开放的挑战，通过元训练L2 O优化器，可以执行快速测试时间自适应的分布任务，只有几个步骤。我们从理论上描述了L2 O的泛化，并进一步表明，我们提出的框架（称为M-L2 O）可证明有利于快速任务自适应定位适应良好的初始点的优化权重。对LASSO和Quadratic等几个经典任务的实证观察表明，M-L2 O的收敛速度明显快于普通L2 O，只需5美元的调整步骤，这与我们的理论结果相呼应。代码可在https://github.com/VITA-Group/M-L2O上找到。
Learning to Optimize (L2O) has drawn increasing attention as it often remarkably accelerates the optimization procedure of complex tasks by ``overfitting"specific task type, leading to enhanced performance compared to analytical optimizers. Generally, L2O develops a parameterized optimization method (i.e., ``optimizer") by learning from solving sample problems. This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same ``task distribution". However, such learned optimizers often struggle when new test problems come with a substantially deviation from the training task distribution. This paper investigates a potential solution to this open challenge, by meta-training an L2O optimizer that can perform fast test-time self-adaptation to an out-of-distribution task, in only a few steps. We theoretically characterize the generalization of L2O, and further show that our proposed framework (termed as M-L2O) provably facilitates rapid task adaptation by locating well-adapted initial points for the optimizer weight. Empirical observations on several classic tasks like LASSO and Quadratic, demonstrate that M-L2O converges significantly faster than vanilla L2O with only $5$ steps of adaptation, echoing our theoretical results. Codes are available in https://github.com/VITA-Group/M-L2O.