Convex Bounds on the Softmax Function with Applications to Robustness Verification
Convex Bounds on the Softmax Function with Applications to Robustness Verification
复制标题
DOI：
10.48550/arxiv.2303.01713
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Dennis L. Wei;Haoze Wu;Min Wu;Pin-Yu Chen;Clark W. Barrett;E. Farchi
中科院分区：
文献类型：
--
作者：
Dennis L. Wei;Haoze Wu;Min Wu;Pin-Yu Chen;Clark W. Barrett;E. Farchi
softmax函数在神经网络的输出端是一个普遍存在的组件，在中间层也越来越多。本文提供了softmax函数的凸下界和凹上界，它们与用于表征神经网络和其他ML模型的凸优化公式兼容。我们使用softmax的自然指数倒数分解以及对数和exp函数的替代分解来推导边界。新的界限是可证明的和/或数值上比线性界限在以前的工作中获得的变压器的鲁棒性验证更严格。作为说明的效用的界限，我们将它们应用到验证变压器以及深度合奏的预测不确定性估计的鲁棒性。
The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.