Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Mahyar Fazlyab;Alexander Robey;Hamed Hassani;M. Morari;George Pappas
中科院分区：
文献类型：
--
作者：
Mahyar Fazlyab;Alexander Robey;Hamed Hassani;M. Morari;George Pappas
深度神经网络 (DNN) 的 Lipschitz 常数的严格估计在许多应用中都很有用，从分类器的鲁棒性认证到具有强化学习控制器的闭环系统的稳定性分析。文献中用于估计 Lipschitz 常数的现有方法要么缺乏准确性，要么可扩展性差。在本文中，我们提出了一个凸优化框架来准确有效地计算 DNN 的 Lipschitz 常数的保证上限。我们的主要思想是将激活函数解释为凸势函数的梯度。因此，它们满足可以通过二次约束描述的某些属性。这种特殊的描述使我们能够将 Lipschitz 常数估计问题提出为半定规划 (SDP)。由此产生的 SDP 可用于提高估计精度（通过捕获不同层激活函数之间的交互）或可扩展性（通过分解和并行实现）。我们通过对随机生成的网络以及在 MNIST 和 Iris 数据集上训练的分类器进行的各种实验来说明我们的方法的实用性。特别是，我们通过实验证明，与文献中的界限相比，我们的 Lipschitz 界限是最准确的。我们还研究了对抗性训练方法对所得分类器的 Lipschitz 边界的影响，并表明我们的边界可用于有效地提供鲁棒性保证。
Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant suffer from either lack of accuracy or poor scalability. In this paper, we present a convex optimization framework to compute guaranteed upper bounds on the Lipschitz constant of DNNs both accurately and efficiently. Our main idea is to interpret activation functions as gradients of convex potential functions. Hence, they satisfy certain properties that can be described by quadratic constraints. This particular description allows us to pose the Lipschitz constant estimation problem as a semidefinite program (SDP). The resulting SDP can be adapted to increase either the estimation accuracy (by capturing the interaction between activation functions of different layers) or scalability (by decomposition and parallel implementation). We illustrate the utility of our approach with a variety of experiments on randomly generated networks and on classifiers trained on the MNIST and Iris datasets. In particular, we experimentally demonstrate that our Lipschitz bounds are the most accurate compared to those in the literature. We also study the impact of adversarial training methods on the Lipschitz bounds of the resulting classifiers and show that our bounds can be used to efficiently provide robustness guarantees.