Precise Tradeoffs in Adversarial Training for Linear Regression
Precise Tradeoffs in Adversarial Training for Linear Regression
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Adel Javanmard;M. Soltanolkotabi;Hamed Hassani
中科院分区：
文献类型：
--
作者：
Adel Javanmard;M. Soltanolkotabi;Hamed Hassani
尽管有突破性的性能，但现代学习模型在输入中非常容易受到小的对抗性扰动的影响。虽然最近各种各样的对抗训练方法在提高对扰动输入的鲁棒性（鲁棒准确性）方面是有效的，但这种好处往往伴随着良性输入的准确性（标准准确性）的降低，导致经常竞争的目标之间的权衡。更复杂的是，最近的经验证据表明，各种其他因素（训练数据的大小和质量，模型大小等）以某种令人惊讶的方式影响着这种权衡。在本文中，我们提供了一个精确和全面的理解对抗训练在具有高斯特征的线性回归背景下的作用。特别是，我们的特点之间的基本权衡的精度可实现的任何算法，无论计算能力或大小的训练数据。此外，我们精确地描述了标准/鲁棒精度以及在高维区域中通过当代最小-最大对抗训练方法实现的相应权衡，其中数据点的数量和模型的参数彼此成比例地增长。我们的对抗性训练算法理论还有助于严格研究各种因素（训练数据的大小和质量，模型过度参数化等）如何影响训练算法。影响这两个竞争精度之间的权衡。
Despite breakthrough performance, modern learning models are known to be highly vulnerable to small adversarial perturbations in their inputs. While a wide variety of recent \emph{adversarial training} methods have been effective at improving robustness to perturbed inputs (robust accuracy), often this benefit is accompanied by a decrease in accuracy on benign inputs (standard accuracy), leading to a tradeoff between often competing objectives. Complicating matters further, recent empirical evidence suggest that a variety of other factors (size and quality of training data, model size, etc.) affect this tradeoff in somewhat surprising ways. In this paper we provide a precise and comprehensive understanding of the role of adversarial training in the context of linear regression with Gaussian features. In particular, we characterize the fundamental tradeoff between the accuracies achievable by any algorithm regardless of computational power or size of the training data. Furthermore, we precisely characterize the standard/robust accuracy and the corresponding tradeoff achieved by a contemporary mini-max adversarial training approach in a high-dimensional regime where the number of data points and the parameters of the model grow in proportion to each other. Our theory for adversarial training algorithms also facilitates the rigorous study of how a variety of factors (size and quality of training data, model overparametrization etc.) affect the tradeoff between these two competing accuracies.