Backpropagation of Unrolled Solvers with Folded Optimization
Backpropagation of Unrolled Solvers with Folded Optimization
复制标题
DOI：
10.24963/ijcai.2023/218
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
James Kotary;M. H. Dinh;Ferdinando Fioretto
中科院分区：
文献类型：
--
作者：
James Kotary;M. H. Dinh;Ferdinando Fioretto
将约束优化模型作为组件集成到深度网络中，在许多专门的学习任务上取得了有希望的进展。这种情况下的主要挑战是通过优化问题的解进行反向传播，而优化问题通常缺乏封闭形式。一种典型的策略是算法展开，它依赖于通过迭代求解器的操作进行自动微分。展开虽然灵活且通用，但在实践中可能会遇到准确性和效率问题。这些问题可以通过优化的分析微分来避免，但目前的框架对优化问题的形式有严格的要求。本文提供了对展开优化的反向传递的理论见解，导致一个系统生成有效可解的反向传播分析模型。此外，它提出了一个统一的视图展开和分析微分通过优化映射。在各种基于模型的学习任务上的实验证明了该方法在计算和增强表达方面的优势。
The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which typically lacks a closed form. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. While flexible and general, unrolling can encounter accuracy and efficiency issues in practice. These issues can be avoided by analytical differentiation of the optimization, but current frameworks impose rigid requirements on the optimization problem's form. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation. Additionally, it proposes a unifying view of unrolling and analytical differentiation through optimization mappings. Experiments over various model-based learning tasks demonstrate the advantages of the approach both computationally and in terms of enhanced expressiveness.