Exploring Machine Learning Privacy/Utility Trade-Off from a Hyperparameters Lens

Exploring Machine Learning Privacy/Utility Trade-Off from a Hyperparameters Lens
复制标题

DOI：
10.1109/ijcnn54540.2023.10191743
复制
发表时间：
2023-03
期刊：
2023 International Joint Conference on Neural Networks (IJCNN)
影响因子：
--
通讯作者：
Ayoub Arous;Amira Guesmi;M. Hanif;Ihsen Alouani;Muhammad Shafique
Ayoub Arous;Amira Guesmi;M. Hanif;Ihsen Alouani;Muhammad Shafique
中科院分区：
其他
文献类型：
--
作者：
Ayoub Arous;Amira Guesmi;M. Hanif;Ihsen Alouani;Muhammad Shafique

文献摘要

相似文献

机器学习（ML）架构已经应用于涉及敏感数据的几个应用程序，其中需要保证用户的数据隐私。差分私密随机梯度下降（DPSGD）是目前最先进的隐私保护模型训练方法。然而，DPSGD的准确性损失相当大，导致在隐私/效用方面的权衡不够理想。为了研究更好的隐私-效用权衡的新领域，这项工作提出了疑问；(i)模型的超参数是否对ML模型的隐私保护属性有任何固有的影响，以及（ii）模型的超参数是否对差异私有模型的隐私/效用权衡有任何影响。我们提出了一个全面的设计空间探索不同的超参数，如激活函数的选择，学习率和批归一化的使用。有趣的是，我们发现通过使用有界的RELU作为具有相同隐私保护特征的激活函数，可以提高效用。通过替换激活函数，我们在不修改DPSGD学习过程基础的情况下，在MNIST（96.02%）、FashionMnist（84.76%）和CIFAR-10（44.42%）上实现了新的最先进的准确率。
Machine Learning (ML) architectures have been applied to several applications that involve sensitive data, where a guarantee of users' data privacy is required. Differentially Private Stochastic Gradient Descent (DPSGD) is the state-of-the-art method to train privacy-preserving models. However, DPSGD comes at a considerable accuracy loss leading to sub-optimal privacy/utility trade-offs. Towards investigating new ground for better privacy-utility trade-off, this work questions; (i) if models' hyperparameters have any inherent impact on ML models' privacy-preserving properties, and (ii) if models' hyperparameters have any impact on the privacy/utility trade-off of differentially private models. We propose a comprehensive design space exploration of different hyperparameters such as the choice of activation functions, the learning rate and the use of batch normalization. Interestingly, we found that utility can be improved by using Bounded RELU as activation functions with the same privacy-preserving characteristics. With a drop-in replacement of the activation function, we achieve new state-of-the-art accuracy on MNIST (96.02%), FashionMnist (84.76%), and CIFAR-10 (44.42%) without any modification of the learning procedure fundamentals of DPSGD.