BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining
BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining
复制标题
DOI：
--
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Weizhe Hua;Yichi Zhang;Chuan Guo;Zhiru Zhang;G. Suh
中科院分区：
文献类型：
--
作者：
Weizhe Hua;Yichi Zhang;Chuan Guo;Zhiru Zhang;G. Suh
近年来，神经网络的鲁棒性已经成为机器学习的一个中心话题。大多数提高模型对对抗性和常见损坏的鲁棒性的训练算法也引入了大量的计算开销，需要多达十倍的向前和向后传递次数才能收敛。为了克服这种低效率，我们提出了BulletTrain $-$边界示例挖掘技术，以大幅降低鲁棒训练的计算成本。我们的关键观察是，只有一小部分的例子是有益的，以提高鲁棒性。BulletTrain动态预测这些重要的示例，并优化强大的训练算法，以专注于重要的示例。我们将我们的技术应用于几个现有的强大的训练算法，并实现了2.1$\times$的加速为TRADES和MART的CIFAR-10和1.7$\times$的加速为AugMix的CIFAR-10-C和CIFAR-100-C没有任何减少清洁和强大的准确性。
Neural network robustness has become a central topic in machine learning in recent years. Most training algorithms that improve the model's robustness to adversarial and common corruptions also introduce a large computational overhead, requiring as many as ten times the number of forward and backward passes in order to converge. To combat this inefficiency, we propose BulletTrain $-$ a boundary example mining technique to drastically reduce the computational cost of robust training. Our key observation is that only a small fraction of examples are beneficial for improving robustness. BulletTrain dynamically predicts these important examples and optimizes robust training algorithms to focus on the important examples. We apply our technique to several existing robust training algorithms and achieve a 2.1$\times$ speed-up for TRADES and MART on CIFAR-10 and a 1.7$\times$ speed-up for AugMix on CIFAR-10-C and CIFAR-100-C without any reduction in clean and robust accuracy.