On the Pitfalls of Security Evaluation of Robust Federated Learning
On the Pitfalls of Security Evaluation of Robust Federated Learning
复制标题
DOI：
10.1109/spw59333.2023.00011
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Momin Ahmad Khan;Virat Shejwalkar;Amir Houmansadr;F. Anwar
中科院分区：
文献类型：
--
作者：
Momin Ahmad Khan;Virat Shejwalkar;Amir Houmansadr;F. Anwar
先前的文献已经证明，联邦学习（FL）容易受到旨在危及FL性能的中毒攻击，因此，已经引入了许多防御措施，并在各种FL设置中证明了它们的鲁棒性。在这项工作中，我们密切调查了一个在很大程度上被忽视的方面，在强大的FL文献，即，用于评估FL中毒防御的鲁棒性的实验设置。我们彻底审查了50防御工程，并强调了FL中毒防御论文的实验设置中的几个值得怀疑的趋势;我们讨论了这些实验设置对这些工程提出的防御鲁棒性的关键结论的潜在影响。作为一个代表性的案例研究，我们还评估了IEEE S& P '23最近的中毒恢复论文，称为FedRecover。我们的案例研究证明了实验设置决策的重要性（例如，例如，虽然FedRecover在MNIST和FashionMNIST（在原始论文中使用）中表现良好，但在我们的实验中，它在FEMNIST和CIFAR 10中表现不佳。
Prior literature has demonstrated that Federated learning (FL) is vulnerable to poisoning attacks that aim to jeopardize FL performance, and consequently, has introduced numerous defenses and demonstrated their robustness in various FL settings. In this work, we closely investigate a largely over-looked aspect in the robust FL literature, i.e., the experimental setup used to evaluate the robustness of FL poisoning defenses. We thoroughly review 50 defense works and highlight several questionable trends in the experimental setup of FL poisoning defense papers; we discuss the potential repercussions of such experimental setups on the key conclusions made by these works about the robustness of the proposed defenses. As a representative case study, we also evaluate a recent poisoning recovery paper from IEEE S&P'23, called FedRecover. Our case study demonstrates the importance of the experimental setup decisions (e.g., selecting representative and challenging datasets) in the validity of the robustness claims; For instance, while FedRecover performs well for MNIST and FashionMNIST (used in the original paper), in our experiments it performed poorly for FEMNIST and CIFAR10.