Optimization-based Implicit Deep Learning, Theory and Applications

过去十年，深度学习取得了显著的成功。然而，当今时代的一个重大挑战是确保这些模型的可解释性和可靠性。在各种应用中，深度神经网络（dnn）需要为其输出提供保证，例如保持自动驾驶汽车在其车道内。另一方面，许多这些任务可以被表述为优化问题，其中优化算法提供可解释和可靠的解决方案。不幸的是，这些模型没有利用数据，因此达不到最先进的深度学习模型。本研究将致力于提高深度学习方法的可解释性和可靠性，并在应用此类学习方法时改善公共安全。此外，该项目将为参与的学生提供宝贵的教育机会。参与者将获得逆问题、优化和机器学习方面的知识，这些都是适用于学术界、政府和工业的可转移技能。该项目旨在开发一个框架，将优化算法的可解释性和可靠性与深度神经网络的设计和训练相结合。主要关注的是隐式网络，这是一种通过定点或最优性条件隐式确定其输出的深度神经网络，而不是像传统深度神经网络那样具有一定数量的层数的固定计算次数。这种将优化算法集成到隐式网络中的方法被称为隐式学习优化（L2O）网络。隐式L2O网络有潜力克服传统深度神经网络的局限性，包括缺乏可靠性和可解释性。然而，训练和设计隐式L2O模型带来了额外的挑战，阻碍了它们的广泛采用。为了应对这些挑战，本研究旨在开发一个通用的隐式L2O框架。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The past decade has seen remarkable success in deep learning. However, a significant challenge in today's era is to ensure interpretability and reliability in these models. In various applications, deep neural networks (DNNs) need to provide guarantees on their outputs, such as maintaining a self-driving car within its lane. On the other hand, many of these tasks can be formulated as optimization problems, where optimization algorithms offer interpretable and reliable solutions. Unfortunately, these models do not leverage data and thus fall short of state-of-the-art deep learning models. This research will address enhancing interpretability and reliability in deep learning methods and improve public safety when such learning methods are applied. In addition, the project will provide valuable educational opportunities for students involved. Participants will gain knowledge in inverse problems, optimization, and machine learning, which are transferable skills applicable in academia, government, and industry. The project aims to develop a framework that combines the interpretability and reliability of optimization algorithms with the design and training of DNNs. The primary focus is on implicit networks, a type of DNNs that determines their outputs implicitly through fixed point or optimality conditions, rather than a fixed number of computations like traditional DNNs with a set number of layers. This integration of optimization algorithms into implicit networks is referred to as implicit learning-to-optimize (L2O) networks. Implicit L2O networks have the potential to overcome the limitations of traditional DNNs, including their lack of reliability and interpretability. However, training and designing implicit L2O models present additional challenges that hinder their widespread adoption. To address these challenges, the research aims to develop a universal implicit L2O framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.